Add Lfm2 support#1510
Open
TensorCruncher wants to merge 8 commits into
Open
Conversation
Contributor
Author
|
Collaborator
|
Hi @TensorCruncher! The PR looks pretty solid so far, I'll be adding a couple notes to the code, but your design choices are sound. Answer to your questions below:
|
Collaborator
|
For the Phase 1 Verification failures:
For the Phase 3 Verification failures: In To fix it, we will want to set Phase 2's yellow |
Contributor
Author
|
I will get to implementing all these changes. Thank you for diagnosing the problems. |
…ts to branch on conv. Ran verify model many times
Contributor
Author
|
Phase 1 is fixed by setting The two flags for phase 3 seem to make no difference. I will post results for plain, + |
Contributor
Author
|
Neither flag set: adil@Syeds-MacBook-Pro TransformerLens % uv run python -m transformer_lens.tools.model_registry.verify_models \
--model LiquidAI/LFM2-1.2B \
--max-memory 16 \
--device cpu \
--dtype float32 \
--no-ht-reference
Model list mode: 1 model(s)
Selected 1 models for verification
2026-07-14 10:49:06,922 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:07,148 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:07,481 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:07,791 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:49:08,007 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:08,301 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-14 10:49:08,729 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-14 10:49:08,730 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 9546.00it/s]
2026-07-14 10:49:09,121 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/generation_config.json "HTTP/1.1 200 OK"
Pre-loaded GPT-2 scoring model for Phase 4
======================================================================
[1/1] LiquidAI/LFM2-1.2B (Lfm2ForCausalLM)
======================================================================
2026-07-14 10:49:09,429 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:09,443 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Estimated parameters: 1,207,959,552
Estimated benchmark memory: 10.8 GB (limit: 16.0 GB)
Running phases [1, 2, 3, 4] in a single benchmark call...
================================================================================
Running TransformerBridge Benchmark Suite
Model: LiquidAI/LFM2-1.2B
Device: cpu
================================================================================
================================================================================
PHASE 1: HuggingFace + TransformerBridge (unprocessed)
================================================================================
Detecting model configuration...
2026-07-14 10:49:09,688 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:09,704 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:27,947 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:27,960 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:28,194 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:28,208 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:28,426 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:49:28,637 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
✓ Detected attn_implementation=eager
Loading HuggingFace reference model...
Using attn_implementation=eager
2026-07-14 10:49:29,397 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:29,411 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:29,639 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:29,653 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:29,917 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:29,931 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 74.26it/s]
2026-07-14 10:49:32,206 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:32,222 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
Detected dtype=torch.float32
✓ HuggingFace model loaded
Loading TransformerBridge (unprocessed)...
2026-07-14 10:49:32,451 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:32,465 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 111.90it/s]
2026-07-14 10:49:34,100 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:34,112 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:34,330 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:34,348 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:34,588 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:34,603 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:49:34,839 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:49:35,054 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
✓ TransformerBridge loaded (unprocessed)
Running Phase 1 benchmarks...
1. Component-Level Benchmarks
🟢 [PASS] all_components: All 114 components produce equivalent outputs
total_components: 114
pass_rate: 100.0
component_types: {'EmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RotaryEmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RMSNormalizationBridge': {'passed': 33, 'failed': 0, 'total': 33}, 'UnembeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'Lfm2ShortConvBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'LinearBridge': {'passed': 42, 'failed': 0, 'total': 42}, 'DepthwiseConv1DBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'GatedMLPBridge': {'passed': 16, 'failed': 0, 'total': 16}}
✓ All 114 components produce equivalent outputs
Capturing HF reference outputs to CPU...
2026-07-14 10:49:43,819 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:43,833 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
✓ Captured HF logits torch.Size([1, 28, 65536]), loss=2.5977
Cleaning up HuggingFace model...
2. Forward Pass Benchmarks
🟢 [PASS] forward_pass_logits: Tensors match within tolerance
atol: 0.001
rtol: 0.03
✓ Saved Phase 1 reference data (logits: torch.Size([1, 28, 65536]))
================================================================================
PHASE 2: TransformerBridge (unprocessed) + HookedTransformer (unprocessed)
================================================================================
Running Phase 2 benchmarks...
2026-07-14 10:49:48,108 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:49:48,123 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
1. Generation Benchmarks (unprocessed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 3.90it/s]
🟢 [PASS] generation: Generation successful: 28 -> 38 tokens (176 -> 248 chars)
input_tokens: 28
output_tokens: 38
input_chars: 176
output_chars: 248
max_new_tokens: 10
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 4.06it/s]
🟢 [PASS] generation_with_kv_cache: KV cache generation successful (231 chars)
output_len: 231
max_new_tokens: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.95it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 3.00it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.97it/s]
🟢 [PASS] multiple_generation_calls: All 3 generation calls successful
prompt_count: 3
max_new_tokens: 5
output_lens: [43, 25, 48]
2. Running Unprocessed Model Comparison Benchmarks
2. Model Equivalence Benchmarks (Forward Pass)
⏭️ Skipped (no HookedTransformer reference)
⏭️ [SKIPPED] logits_equivalence: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] loss_equivalence: Skipped (HookedTransformer not available for this model)
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 588 registered hooks
hook_count: 588
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 78/787 hooks didn't fire during forward pass
captured: 709
registered: 787
didnt_fire: ['blocks.5.attn.hook_q_input', 'blocks.10.attn.hook_attn_in', 'blocks.12.attn.hook_k_input', 'blocks.3.hook_mlp_in', 'blocks.14.attn.hook_hidden_states', 'blocks.2.attn.hook_attn_in', 'blocks.8.attn.hook_hidden_states', 'blocks.13.hook_mlp_in', 'blocks.7.hook_mlp_in', 'blocks.8.attn.hook_k_input']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 709 cached activations
cache_size: 709
🟢 [PASS] activation_cache: Activation cache created with 709 entries
cache_size: 709
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 429 backward hook gradients
gradient_count: 429
2026-07-14 10:50:18,029 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:50:18,044 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:50:18,295 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:50:18,310 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 2.5: Text Quality (GPT-2 perplexity scoring)
================================================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:10<00:00, 4.82it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:10<00:00, 4.91it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.01it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:10<00:00, 4.86it/s]
🟢 [PASS] text_quality: Text quality score: 99.3/100 (avg perplexity: 22.9, 4 prompts)
score: 99.3
avg_perplexity: 22.87
avg_repetition_penalty: 0.99
num_prompts: 4
per_prompt: ppl=15.6 score=100.0 rep=1.00 | ppl=9.9 score=97.6 rep=0.98 | ppl=34.7 score=99.5 rep=1.00 | ppl=31.3 score=100.0 rep=1.00
scoring_model: gpt2
max_new_tokens: 50
generated_text: Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning. However, these tasks often require large amounts of labeled data, which can be hard to come by, especially for niche domains. Self-supervised learning (SSL) has emerged as a powerful technique to address this challenge by leveraging unlabeled data to
2026-07-14 10:51:00,733 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:51:00,746 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 3: TransformerBridge (processed) + HookedTransformer (processed)
================================================================================
Processing weights on existing bridge (reusing Phase 1 instance)...
✓ TransformerBridge compatibility mode enabled (processed)
Running Phase 3 benchmarks...
1. Weight Processing Benchmarks (Foundation)
🟢 [PASS] no_nan_inf: No NaN or Inf values found in weights
num_tensors_checked: 150
🟢 [PASS] weight_magnitudes: All weight magnitudes in reasonable ranges
min_threshold: 1e-06
max_threshold: 1000.0
🟡 [PASS] layer_norm_folding: Partial LN folding: ln2 (mean=1.000000) folded; ln1 (mean=0.235300) preserved (expected for combined QKV models)
folded: ['ln2']
not_folded: ['ln1']
🟢 [PASS] attention_output_centering: Attention output centering verified (worst_mean=0.000390) (6/16 blocks have attention)
mean: 0.00039015396032482386
tolerance: 0.01
n_attn_blocks: 6
🟢 [PASS] mlp_output_centering: MLP output centering verified (mean=0.000185)
mean: 0.00018529227236285806
tolerance: 0.01
🟢 [PASS] unembed_centering: Unembed centering verified (mean=0.000000)
mean: 8.756500524675914e-10
tolerance: 0.01
key: unembed.weight
🟢 [PASS] value_bias_folding: Skipped for GQA models (n_key_value_heads != n_heads)
is_gqa: True
n_heads: 32
n_kv_heads: 8
⏭️ weight_processing and weight_sharing skipped (no HT reference)
⏭️ [SKIPPED] weight_processing: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] weight_sharing: Skipped (HookedTransformer not available for this model)
🟢 [PASS] weight_modification: Weight modification propagates correctly (change: 0.101187)
change: 0.10118675231933594
2. Model Equivalence Benchmarks (Forward Pass)
Using saved Phase 1 bridge reference for equivalence comparison
🔴 [FAIL] logits_equivalence: Tensors differ: max_diff=23.349289, mean_rel=0.575189
max_diff: 23.349288940429688
mean_diff: 6.9689178466796875
mean_rel: 0.5751886367797852
atol: 0.01
rtol: 0.0001
🔴 [FAIL] loss_equivalence: Scalars differ: 12.080062 vs 2.597675
diff: 9.48238730430603
atol: 0.001
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 787 registered hooks
hook_count: 787
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 126/787 hooks didn't fire during forward pass
captured: 661
registered: 787
didnt_fire: ['blocks.5.attn.hook_q_input', 'blocks.10.attn.hook_attn_in', 'blocks.6.mlp.in.hook_in', 'blocks.12.attn.hook_k_input', 'blocks.3.hook_mlp_in', 'blocks.15.mlp.out.hook_out', 'blocks.3.mlp.in.hook_in', 'blocks.14.attn.hook_hidden_states', 'blocks.11.mlp.out.hook_out', 'blocks.2.attn.hook_attn_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 661 cached activations
cache_size: 661
🟢 [PASS] activation_cache: Activation cache created with 661 entries
cache_size: 661
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 555 backward hook gradients
gradient_count: 555
Cleaning up TransformerBridge (processed)...
================================================================================
BENCHMARK SUMMARY
================================================================================
Phase 1: 2 tests run
Passed: 2/2 (100.0%)
Failed: 0/2 (0.0%)
Phase 2: 13 tests run
Passed: 13/13 (100.0%)
Failed: 0/13 (0.0%)
Skipped: 2
Phase 3: 20 tests run
Passed: 18/20 (90.0%)
Failed: 2/20 (10.0%)
Skipped: 2
Phase 4: 1 tests run
Passed: 1/1 (100.0%)
Failed: 0/1 (0.0%)
Overall:
Total: 40 tests
Run: 36 tests
Skipped: 4 tests
Passed: 34/36 (94.4%)
Failed: 2/36 (5.6%)
================================================================================
FAILED: Below threshold: P3=90.0% but required tests failed: logits_equivalence, loss_equivalence — Tensors differ: max_diff=23.349289, mean_rel=0.575189
Partial scores saved: P1=100.0%, P2=100.0%, P3=90.0%, P4=99.3%, P7=None%, P8=None%
======================================================================
Verification Summary
======================================================================
Total tested: 1
Verified: 0
Skipped: 0
Failed: 1
Failed models:
- LiquidAI/LFM2-1.2B
Total time: 147.5s
Checkpoint cleared (run complete)
adil@Syeds-MacBook-Pro TransformerLens %
adil@Syeds-MacBook-Pro TransformerLens %
|
Contributor
Author
|
self.supports_fold_ln = False adil@Syeds-MacBook-Pro TransformerLens % uv run python -m transformer_lens.tools.model_registry.verify_models \
--model LiquidAI/LFM2-1.2B \
--max-memory 16 \
--device cpu \
--dtype float32 \
--no-ht-reference
Model list mode: 1 model(s)
Selected 1 models for verification
2026-07-14 10:54:53,996 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-14 10:54:53,997 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-14 10:54:54,229 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:54:54,443 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:54:54,663 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:54:54,917 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:54:55,150 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-14 10:54:55,639 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 23244.97it/s]
2026-07-14 10:54:56,351 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/generation_config.json "HTTP/1.1 200 OK"
Pre-loaded GPT-2 scoring model for Phase 4
======================================================================
[1/1] LiquidAI/LFM2-1.2B (Lfm2ForCausalLM)
======================================================================
2026-07-14 10:54:56,657 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:54:56,673 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Estimated parameters: 1,207,959,552
Estimated benchmark memory: 10.8 GB (limit: 16.0 GB)
Running phases [1, 2, 3, 4] in a single benchmark call...
================================================================================
Running TransformerBridge Benchmark Suite
Model: LiquidAI/LFM2-1.2B
Device: cpu
================================================================================
================================================================================
PHASE 1: HuggingFace + TransformerBridge (unprocessed)
================================================================================
Detecting model configuration...
2026-07-14 10:54:56,919 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:54:56,932 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:15,578 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:15,592 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:15,822 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:15,835 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:16,072 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:55:16,299 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
✓ Detected attn_implementation=eager
Loading HuggingFace reference model...
Using attn_implementation=eager
2026-07-14 10:55:17,015 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:17,035 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:17,272 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:17,284 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:17,520 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:17,536 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 117.81it/s]
2026-07-14 10:55:19,110 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:19,126 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
Detected dtype=torch.float32
✓ HuggingFace model loaded
Loading TransformerBridge (unprocessed)...
2026-07-14 10:55:19,370 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:19,383 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 93.67it/s]
2026-07-14 10:55:21,351 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:21,365 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:21,581 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:21,595 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:21,825 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:21,838 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 10:55:22,084 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 10:55:22,312 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
✓ TransformerBridge loaded (unprocessed)
Running Phase 1 benchmarks...
1. Component-Level Benchmarks
🟢 [PASS] all_components: All 114 components produce equivalent outputs
total_components: 114
pass_rate: 100.0
component_types: {'EmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RotaryEmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RMSNormalizationBridge': {'passed': 33, 'failed': 0, 'total': 33}, 'UnembeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'Lfm2ShortConvBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'LinearBridge': {'passed': 42, 'failed': 0, 'total': 42}, 'DepthwiseConv1DBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'GatedMLPBridge': {'passed': 16, 'failed': 0, 'total': 16}}
✓ All 114 components produce equivalent outputs
Capturing HF reference outputs to CPU...
2026-07-14 10:55:32,173 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:32,188 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
✓ Captured HF logits torch.Size([1, 28, 65536]), loss=2.5977
Cleaning up HuggingFace model...
2. Forward Pass Benchmarks
🟢 [PASS] forward_pass_logits: Tensors match within tolerance
atol: 0.001
rtol: 0.03
✓ Saved Phase 1 reference data (logits: torch.Size([1, 28, 65536]))
================================================================================
PHASE 2: TransformerBridge (unprocessed) + HookedTransformer (unprocessed)
================================================================================
Running Phase 2 benchmarks...
2026-07-14 10:55:36,603 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:55:36,617 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
1. Generation Benchmarks (unprocessed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 3.87it/s]
🟢 [PASS] generation: Generation successful: 28 -> 38 tokens (176 -> 231 chars)
input_tokens: 28
output_tokens: 38
input_chars: 176
output_chars: 231
max_new_tokens: 10
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 4.03it/s]
🟢 [PASS] generation_with_kv_cache: KV cache generation successful (237 chars)
output_len: 237
max_new_tokens: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.99it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.94it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.96it/s]
🟢 [PASS] multiple_generation_calls: All 3 generation calls successful
prompt_count: 3
max_new_tokens: 5
output_lens: [43, 25, 44]
2. Running Unprocessed Model Comparison Benchmarks
2. Model Equivalence Benchmarks (Forward Pass)
⏭️ Skipped (no HookedTransformer reference)
⏭️ [SKIPPED] logits_equivalence: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] loss_equivalence: Skipped (HookedTransformer not available for this model)
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 588 registered hooks
hook_count: 588
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 78/787 hooks didn't fire during forward pass
captured: 709
registered: 787
didnt_fire: ['blocks.14.attn.hook_attn_in', 'blocks.14.attn.hook_result', 'blocks.5.attn.hook_v_input', 'blocks.1.hook_mlp_in', 'blocks.0.hook_mlp_in', 'blocks.8.hook_attn_in', 'blocks.8.attn.hook_result', 'blocks.12.hook_k_input', 'blocks.12.attn.hook_q_input', 'blocks.8.attn.hook_hidden_states']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 709 cached activations
cache_size: 709
🟢 [PASS] activation_cache: Activation cache created with 709 entries
cache_size: 709
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 429 backward hook gradients
gradient_count: 429
2026-07-14 10:56:05,913 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:56:05,924 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 10:56:06,151 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:56:06,167 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 2.5: Text Quality (GPT-2 perplexity scoring)
================================================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.22it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.18it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.25it/s]
🟢 [PASS] text_quality: Text quality score: 99.3/100 (avg perplexity: 22.9, 4 prompts)
score: 99.3
avg_perplexity: 22.87
avg_repetition_penalty: 0.99
num_prompts: 4
per_prompt: ppl=15.6 score=100.0 rep=1.00 | ppl=9.9 score=97.6 rep=0.98 | ppl=34.7 score=99.5 rep=1.00 | ppl=31.3 score=100.0 rep=1.00
scoring_model: gpt2
max_new_tokens: 50
generated_text: Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning. However, these tasks often require large amounts of labeled data, which can be hard to come by, especially for niche domains. Self-supervised learning (SSL) has emerged as a powerful technique to address this challenge by leveraging unlabeled data to
2026-07-14 10:56:45,077 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 10:56:45,090 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 3: TransformerBridge (processed) + HookedTransformer (processed)
================================================================================
Processing weights on existing bridge (reusing Phase 1 instance)...
✓ TransformerBridge compatibility mode enabled (processed)
Running Phase 3 benchmarks...
1. Weight Processing Benchmarks (Foundation)
🟢 [PASS] no_nan_inf: No NaN or Inf values found in weights
num_tensors_checked: 150
🟢 [PASS] weight_magnitudes: All weight magnitudes in reasonable ranges
min_threshold: 1e-06
max_threshold: 1000.0
⏭️ [SKIPPED] layer_norm_folding: Skipped (post-LN architecture does not support fold_ln)
🟢 [PASS] attention_output_centering: Attention output centering verified (worst_mean=0.000390) (6/16 blocks have attention)
mean: 0.00039015396032482386
tolerance: 0.01
n_attn_blocks: 6
🟢 [PASS] mlp_output_centering: MLP output centering verified (mean=0.000185)
mean: 0.00018529227236285806
tolerance: 0.01
🟢 [PASS] unembed_centering: Unembed centering verified (mean=0.000000)
mean: 4.757514382447425e-10
tolerance: 0.01
key: unembed.weight
🟢 [PASS] value_bias_folding: Skipped for GQA models (n_key_value_heads != n_heads)
is_gqa: True
n_heads: 32
n_kv_heads: 8
⏭️ weight_processing and weight_sharing skipped (no HT reference)
⏭️ [SKIPPED] weight_processing: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] weight_sharing: Skipped (HookedTransformer not available for this model)
🟢 [PASS] weight_modification: Weight modification propagates correctly (change: 0.101190)
change: 0.10118961334228516
2. Model Equivalence Benchmarks (Forward Pass)
Using saved Phase 1 bridge reference for equivalence comparison
🔴 [FAIL] logits_equivalence: Tensors differ: max_diff=23.349285, mean_rel=0.575189
max_diff: 23.349285125732422
mean_diff: 6.9689178466796875
mean_rel: 0.5751886367797852
atol: 0.01
rtol: 0.0001
🔴 [FAIL] loss_equivalence: Scalars differ: 12.080060 vs 2.597675
diff: 9.482385396957397
atol: 0.001
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 787 registered hooks
hook_count: 787
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 126/787 hooks didn't fire during forward pass
captured: 661
registered: 787
didnt_fire: ['blocks.14.attn.hook_attn_in', 'blocks.0.mlp.out.hook_out', 'blocks.13.mlp.in.hook_in', 'blocks.6.mlp.in.hook_in', 'blocks.14.attn.hook_result', 'blocks.10.mlp.out.hook_out', 'blocks.5.attn.hook_v_input', 'blocks.1.hook_mlp_in', 'blocks.9.mlp.out.hook_out', 'blocks.0.hook_mlp_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 661 cached activations
cache_size: 661
🟢 [PASS] activation_cache: Activation cache created with 661 entries
cache_size: 661
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 555 backward hook gradients
gradient_count: 555
Cleaning up TransformerBridge (processed)...
================================================================================
BENCHMARK SUMMARY
================================================================================
Phase 1: 2 tests run
Passed: 2/2 (100.0%)
Failed: 0/2 (0.0%)
Phase 2: 13 tests run
Passed: 13/13 (100.0%)
Failed: 0/13 (0.0%)
Skipped: 2
Phase 3: 19 tests run
Passed: 17/19 (89.5%)
Failed: 2/19 (10.5%)
Skipped: 3
Phase 4: 1 tests run
Passed: 1/1 (100.0%)
Failed: 0/1 (0.0%)
Overall:
Total: 40 tests
Run: 35 tests
Skipped: 5 tests
Passed: 33/35 (94.3%)
Failed: 2/35 (5.7%)
================================================================================
FAILED: Below threshold: P3=89.5% but required tests failed: logits_equivalence, loss_equivalence — Tensors differ: max_diff=23.349285, mean_rel=0.575189
Partial scores saved: P1=100.0%, P2=100.0%, P3=89.5%, P4=99.3%, P7=None%, P8=None%
======================================================================
Verification Summary
======================================================================
Total tested: 1
Verified: 0
Skipped: 0
Failed: 1
Failed models:
- LiquidAI/LFM2-1.2B
Total time: 142.7s
Checkpoint cleared (run complete)
adil@Syeds-MacBook-Pro TransformerLens %
|
Contributor
Author
|
self.supports_fold_ln = False adil@Syeds-MacBook-Pro TransformerLens % uv run python -m transformer_lens.tools.model_registry.verify_models \
--model LiquidAI/LFM2-1.2B \
--max-memory 16 \
--device cpu \
--dtype float32 \
--no-ht-reference
Model list mode: 1 model(s)
Selected 1 models for verification
2026-07-14 11:01:52,804 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-14 11:01:52,805 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-14 11:01:53,021 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 11:01:53,253 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:01:53,487 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 11:01:53,720 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:01:53,953 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-14 11:01:54,320 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 35663.39it/s]
2026-07-14 11:01:54,657 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/generation_config.json "HTTP/1.1 200 OK"
Pre-loaded GPT-2 scoring model for Phase 4
======================================================================
[1/1] LiquidAI/LFM2-1.2B (Lfm2ForCausalLM)
======================================================================
2026-07-14 11:01:54,871 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:01:54,883 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Estimated parameters: 1,207,959,552
Estimated benchmark memory: 10.8 GB (limit: 16.0 GB)
Running phases [1, 2, 3, 4] in a single benchmark call...
================================================================================
Running TransformerBridge Benchmark Suite
Model: LiquidAI/LFM2-1.2B
Device: cpu
================================================================================
================================================================================
PHASE 1: HuggingFace + TransformerBridge (unprocessed)
================================================================================
Detecting model configuration...
2026-07-14 11:01:55,115 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:01:55,128 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:13,383 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:13,396 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:13,640 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:13,656 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:13,916 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 11:02:14,217 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
✓ Detected attn_implementation=eager
Loading HuggingFace reference model...
Using attn_implementation=eager
2026-07-14 11:02:14,936 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:14,951 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:15,168 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:15,190 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:15,414 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:15,432 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 147.03it/s]
2026-07-14 11:02:16,712 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:16,728 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
Detected dtype=torch.float32
✓ HuggingFace model loaded
Loading TransformerBridge (unprocessed)...
2026-07-14 11:02:16,960 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:16,981 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 106.55it/s]
2026-07-14 11:02:18,721 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:18,734 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:18,947 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:18,962 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:19,199 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:19,218 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-14 11:02:19,450 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-14 11:02:19,696 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
✓ TransformerBridge loaded (unprocessed)
Running Phase 1 benchmarks...
1. Component-Level Benchmarks
🟢 [PASS] all_components: All 114 components produce equivalent outputs
total_components: 114
pass_rate: 100.0
component_types: {'EmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RotaryEmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RMSNormalizationBridge': {'passed': 33, 'failed': 0, 'total': 33}, 'UnembeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'Lfm2ShortConvBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'LinearBridge': {'passed': 42, 'failed': 0, 'total': 42}, 'DepthwiseConv1DBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'GatedMLPBridge': {'passed': 16, 'failed': 0, 'total': 16}}
✓ All 114 components produce equivalent outputs
Capturing HF reference outputs to CPU...
2026-07-14 11:02:29,575 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:29,590 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
✓ Captured HF logits torch.Size([1, 28, 65536]), loss=2.5977
Cleaning up HuggingFace model...
2. Forward Pass Benchmarks
🟢 [PASS] forward_pass_logits: Tensors match within tolerance
atol: 0.001
rtol: 0.03
✓ Saved Phase 1 reference data (logits: torch.Size([1, 28, 65536]))
================================================================================
PHASE 2: TransformerBridge (unprocessed) + HookedTransformer (unprocessed)
================================================================================
Running Phase 2 benchmarks...
2026-07-14 11:02:33,796 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:02:33,812 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
1. Generation Benchmarks (unprocessed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 3.88it/s]
🟢 [PASS] generation: Generation successful: 28 -> 38 tokens (176 -> 239 chars)
input_tokens: 28
output_tokens: 38
input_chars: 176
output_chars: 239
max_new_tokens: 10
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 4.05it/s]
🟢 [PASS] generation_with_kv_cache: KV cache generation successful (242 chars)
output_len: 242
max_new_tokens: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.99it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 3.00it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.98it/s]
🟢 [PASS] multiple_generation_calls: All 3 generation calls successful
prompt_count: 3
max_new_tokens: 5
output_lens: [43, 26, 45]
2. Running Unprocessed Model Comparison Benchmarks
2. Model Equivalence Benchmarks (Forward Pass)
⏭️ Skipped (no HookedTransformer reference)
⏭️ [SKIPPED] logits_equivalence: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] loss_equivalence: Skipped (HookedTransformer not available for this model)
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 588 registered hooks
hook_count: 588
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 78/787 hooks didn't fire during forward pass
captured: 709
registered: 787
didnt_fire: ['blocks.2.hook_v_input', 'rotary_emb.hook_out', 'blocks.2.attn.hook_result', 'blocks.12.hook_attn_in', 'blocks.2.attn.hook_v_input', 'blocks.5.hook_v_input', 'blocks.2.attn.hook_attn_in', 'blocks.5.attn.hook_q_input', 'blocks.14.hook_mlp_in', 'blocks.10.hook_attn_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 709 cached activations
cache_size: 709
🟢 [PASS] activation_cache: Activation cache created with 709 entries
cache_size: 709
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 429 backward hook gradients
gradient_count: 429
2026-07-14 11:03:04,195 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:03:04,213 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-14 11:03:04,446 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:03:04,461 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 2.5: Text Quality (GPT-2 perplexity scoring)
================================================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.24it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.34it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.33it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.30it/s]
🟢 [PASS] text_quality: Text quality score: 99.3/100 (avg perplexity: 22.9, 4 prompts)
score: 99.3
avg_perplexity: 22.87
avg_repetition_penalty: 0.99
num_prompts: 4
per_prompt: ppl=15.6 score=100.0 rep=1.00 | ppl=9.9 score=97.6 rep=0.98 | ppl=34.7 score=99.5 rep=1.00 | ppl=31.3 score=100.0 rep=1.00
scoring_model: gpt2
max_new_tokens: 50
generated_text: Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning. However, these tasks often require large amounts of labeled data, which can be hard to come by, especially for niche domains. Self-supervised learning (SSL) has emerged as a powerful technique to address this challenge by leveraging unlabeled data to
2026-07-14 11:03:42,791 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-14 11:03:42,805 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 3: TransformerBridge (processed) + HookedTransformer (processed)
================================================================================
Processing weights on existing bridge (reusing Phase 1 instance)...
✓ TransformerBridge compatibility mode enabled (processed)
Running Phase 3 benchmarks...
1. Weight Processing Benchmarks (Foundation)
🟢 [PASS] no_nan_inf: No NaN or Inf values found in weights
num_tensors_checked: 150
🟢 [PASS] weight_magnitudes: All weight magnitudes in reasonable ranges
min_threshold: 1e-06
max_threshold: 1000.0
⏭️ [SKIPPED] layer_norm_folding: Skipped (post-LN architecture does not support fold_ln)
🟢 [PASS] attention_output_centering: Attention output centering verified (worst_mean=0.000390) (6/16 blocks have attention)
mean: 0.00039015396032482386
tolerance: 0.01
n_attn_blocks: 6
🟢 [PASS] mlp_output_centering: MLP output centering verified (mean=0.000185)
mean: 0.00018529227236285806
tolerance: 0.01
🟢 [PASS] unembed_centering: Unembed centering verified (mean=0.000000)
mean: 4.757514382447425e-10
tolerance: 0.01
key: unembed.weight
🟢 [PASS] value_bias_folding: Skipped for GQA models (n_key_value_heads != n_heads)
is_gqa: True
n_heads: 32
n_kv_heads: 8
⏭️ weight_processing and weight_sharing skipped (no HT reference)
⏭️ [SKIPPED] weight_processing: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] weight_sharing: Skipped (HookedTransformer not available for this model)
🟢 [PASS] weight_modification: Weight modification propagates correctly (change: 0.101190)
change: 0.10118961334228516
2. Model Equivalence Benchmarks (Forward Pass)
Using saved Phase 1 bridge reference for equivalence comparison
🔴 [FAIL] logits_equivalence: Tensors differ: max_diff=23.349285, mean_rel=0.575189
max_diff: 23.349285125732422
mean_diff: 6.9689178466796875
mean_rel: 0.5751886367797852
atol: 0.01
rtol: 0.0001
🔴 [FAIL] loss_equivalence: Scalars differ: 12.080060 vs 2.597675
diff: 9.482385396957397
atol: 0.001
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 787 registered hooks
hook_count: 787
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 126/787 hooks didn't fire during forward pass
captured: 661
registered: 787
didnt_fire: ['blocks.2.hook_v_input', 'rotary_emb.hook_out', 'blocks.2.attn.hook_result', 'blocks.12.hook_attn_in', 'blocks.14.mlp.out.hook_out', 'blocks.10.mlp.out.hook_out', 'blocks.3.mlp.in.hook_in', 'blocks.2.attn.hook_v_input', 'blocks.5.hook_v_input', 'blocks.2.attn.hook_attn_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 661 cached activations
cache_size: 661
🟢 [PASS] activation_cache: Activation cache created with 661 entries
cache_size: 661
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 555 backward hook gradients
gradient_count: 555
Cleaning up TransformerBridge (processed)...
================================================================================
BENCHMARK SUMMARY
================================================================================
Phase 1: 2 tests run
Passed: 2/2 (100.0%)
Failed: 0/2 (0.0%)
Phase 2: 13 tests run
Passed: 13/13 (100.0%)
Failed: 0/13 (0.0%)
Skipped: 2
Phase 3: 19 tests run
Passed: 17/19 (89.5%)
Failed: 2/19 (10.5%)
Skipped: 3
Phase 4: 1 tests run
Passed: 1/1 (100.0%)
Failed: 0/1 (0.0%)
Overall:
Total: 40 tests
Run: 35 tests
Skipped: 5 tests
Passed: 33/35 (94.3%)
Failed: 2/35 (5.7%)
================================================================================
FAILED: Below threshold: P3=89.5% but required tests failed: logits_equivalence, loss_equivalence — Tensors differ: max_diff=23.349285, mean_rel=0.575189
Partial scores saved: P1=100.0%, P2=100.0%, P3=89.5%, P4=99.3%, P7=None%, P8=None%
======================================================================
Verification Summary
======================================================================
Total tested: 1
Verified: 0
Skipped: 0
Failed: 1
Failed models:
- LiquidAI/LFM2-1.2B
Total time: 140.0s
Checkpoint cleared (run complete)
adil@Syeds-MacBook-Pro TransformerLens %
|
Collaborator
|
Tracked down the Phase 3 failure locally on LFM2.5-230M (your flags ablation was the key clue. Bit-identical failures meant the corruption was flag-independent). From what I can tell, it's the Two smaller notes: I believe the solution will be in the |
1 task
Collaborator
|
@TensorCruncher Just merged the latest |
Contributor
Author
|
I'll implement these changes and re run verification. |
Contributor
Author
|
So weight processing pipeline already appears to untie the weights. The error between unprocessed and processed TransformerBridge instances seems to start around the mlp in layer 0. Checking the If you'd like, you can merge this PR. I can make separate PRs for the supports ln fold gating and the integration test for this adapter (is the integration test for this adapter or for the new lfm2bridge element?). Or i could add further commits to this PR. Let me know! |
Contributor
Author
adil@Syeds-MacBook-Pro TransformerLens % uv run python -m transformer_lens.tools.model_registry.verify_models \
--model LiquidAI/LFM2-1.2B \
--max-memory 16 \
--device cpu \
--dtype float32 \
--no-ht-reference
Model list mode: 1 model(s)
Selected 1 models for verification
2026-07-15 20:17:22,556 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-15 20:17:22,557 [WARNING] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-07-15 20:17:22,816 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:23,075 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:23,557 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-15 20:17:23,865 [INFO] HTTP Request: GET https://huggingface.co/api/models/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:24,102 [INFO] HTTP Request: GET https://huggingface.co/api/models/openai-community/gpt2/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-07-15 20:17:24,471 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:00<00:00, 8800.07it/s]
2026-07-15 20:17:25,093 [INFO] HTTP Request: HEAD https://huggingface.co/gpt2/resolve/main/generation_config.json "HTTP/1.1 200 OK"
Pre-loaded GPT-2 scoring model for Phase 4
======================================================================
[1/1] LiquidAI/LFM2-1.2B (Lfm2ForCausalLM)
======================================================================
2026-07-15 20:17:25,362 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:25,397 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Estimated parameters: 1,207,959,552
Estimated benchmark memory: 10.8 GB (limit: 16.0 GB)
Running phases [1, 2, 3, 4] in a single benchmark call...
================================================================================
Running TransformerBridge Benchmark Suite
Model: LiquidAI/LFM2-1.2B
Device: cpu
================================================================================
================================================================================
PHASE 1: HuggingFace + TransformerBridge (unprocessed)
================================================================================
Detecting model configuration...
2026-07-15 20:17:25,646 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:25,692 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:44,097 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:44,128 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:44,379 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:44,418 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:44,691 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-15 20:17:44,940 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.0') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.1') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.3') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.4') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.6') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.7') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.9') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.11') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.13') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_in' -> 'attn.hook_attn_in' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_attn_out' -> 'attn.hook_out' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_q_input' -> 'attn.hook_q_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_k_input' -> 'attn.hook_k_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
/Users/adil/Desktop/Open_Source/TransformerLens/transformer_lens/model_bridge/bridge.py:489: UserWarning: Hook alias 'hook_v_input' -> 'attn.hook_v_input' on BlockBridge(name='model.layers.15') did not resolve; this hook will not be accessible.
getattr(module, "_register_aliases")()
✓ Detected attn_implementation=eager
Loading HuggingFace reference model...
Using attn_implementation=eager
2026-07-15 20:17:45,647 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:45,687 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:45,950 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:45,987 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:46,232 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:46,281 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:02<00:00, 62.49it/s]
2026-07-15 20:17:48,961 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:49,007 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
Detected dtype=torch.float32
✓ HuggingFace model loaded
Loading TransformerBridge (unprocessed)...
2026-07-15 20:17:49,287 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:49,327 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 148/148 [00:01<00:00, 90.89it/s]
2026-07-15 20:17:51,313 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:51,351 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/generation_config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:51,591 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:51,630 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:51,871 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:17:51,909 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/tokenizer_config.json "HTTP/1.1 200 OK"
2026-07-15 20:17:52,160 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-07-15 20:17:52,434 [INFO] HTTP Request: GET https://huggingface.co/api/models/LiquidAI/LFM2-1.2B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
✓ TransformerBridge loaded (unprocessed)
Running Phase 1 benchmarks...
1. Component-Level Benchmarks
🟢 [PASS] all_components: All 114 components produce equivalent outputs
total_components: 114
pass_rate: 100.0
component_types: {'EmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RotaryEmbeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'RMSNormalizationBridge': {'passed': 33, 'failed': 0, 'total': 33}, 'UnembeddingBridge': {'passed': 1, 'failed': 0, 'total': 1}, 'Lfm2ShortConvBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'LinearBridge': {'passed': 42, 'failed': 0, 'total': 42}, 'DepthwiseConv1DBridge': {'passed': 10, 'failed': 0, 'total': 10}, 'GatedMLPBridge': {'passed': 16, 'failed': 0, 'total': 16}}
✓ All 114 components produce equivalent outputs
Capturing HF reference outputs to CPU...
2026-07-15 20:18:02,098 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:18:02,135 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
✓ Captured HF logits torch.Size([1, 28, 65536]), loss=2.5977
Cleaning up HuggingFace model...
2. Forward Pass Benchmarks
🟢 [PASS] forward_pass_logits: Tensors match within tolerance
atol: 0.001
rtol: 0.03
✓ Saved Phase 1 reference data (logits: torch.Size([1, 28, 65536]))
================================================================================
PHASE 2: TransformerBridge (unprocessed) + HookedTransformer (unprocessed)
================================================================================
Running Phase 2 benchmarks...
2026-07-15 20:18:06,912 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:18:06,953 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
1. Generation Benchmarks (unprocessed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 4.04it/s]
🟢 [PASS] generation: Generation successful: 28 -> 38 tokens (176 -> 226 chars)
input_tokens: 28
output_tokens: 38
input_chars: 176
output_chars: 226
max_new_tokens: 10
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00, 4.08it/s]
🟢 [PASS] generation_with_kv_cache: KV cache generation successful (238 chars)
output_len: 238
max_new_tokens: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 2.99it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 3.01it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00, 3.00it/s]
🟢 [PASS] multiple_generation_calls: All 3 generation calls successful
prompt_count: 3
max_new_tokens: 5
output_lens: [43, 24, 51]
2. Running Unprocessed Model Comparison Benchmarks
2. Model Equivalence Benchmarks (Forward Pass)
⏭️ Skipped (no HookedTransformer reference)
⏭️ [SKIPPED] logits_equivalence: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] loss_equivalence: Skipped (HookedTransformer not available for this model)
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 588 registered hooks
hook_count: 588
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 78/787 hooks didn't fire during forward pass
captured: 709
registered: 787
didnt_fire: ['blocks.1.hook_mlp_in', 'blocks.5.hook_attn_in', 'blocks.5.attn.hook_attn_in', 'blocks.11.hook_mlp_in', 'blocks.5.hook_mlp_in', 'blocks.8.attn.hook_attn_in', 'blocks.8.hook_mlp_in', 'blocks.10.attn.hook_result', 'blocks.9.hook_mlp_in', 'blocks.2.hook_attn_in']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 709 cached activations
cache_size: 709
🟢 [PASS] activation_cache: Activation cache created with 709 entries
cache_size: 709
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 429 backward hook gradients
gradient_count: 429
2026-07-15 20:18:37,521 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:18:37,555 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
2026-07-15 20:18:37,801 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:18:37,834 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 2.5: Text Quality (GPT-2 perplexity scoring)
================================================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.26it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.26it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00, 5.24it/s]
🟢 [PASS] text_quality: Text quality score: 99.3/100 (avg perplexity: 22.9, 4 prompts)
score: 99.3
avg_perplexity: 22.87
avg_repetition_penalty: 0.99
num_prompts: 4
per_prompt: ppl=15.6 score=100.0 rep=1.00 | ppl=9.9 score=97.6 rep=0.98 | ppl=34.7 score=99.5 rep=1.00 | ppl=31.3 score=100.0 rep=1.00
scoring_model: gpt2
max_new_tokens: 50
generated_text: Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning. However, these tasks often require large amounts of labeled data, which can be hard to come by, especially for niche domains. Self-supervised learning (SSL) has emerged as a powerful technique to address this challenge by leveraging unlabeled data to
2026-07-15 20:19:17,614 [INFO] HTTP Request: HEAD https://huggingface.co/LiquidAI/LFM2-1.2B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-07-15 20:19:17,646 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/LiquidAI/LFM2-1.2B/933cee00d754fb3bfe06c644c0cb95453f2d8bb2/config.json "HTTP/1.1 200 OK"
================================================================================
PHASE 3: TransformerBridge (processed) + HookedTransformer (processed)
================================================================================
Processing weights on existing bridge (reusing Phase 1 instance)...
✓ TransformerBridge compatibility mode enabled (processed)
Running Phase 3 benchmarks...
1. Weight Processing Benchmarks (Foundation)
🟢 [PASS] no_nan_inf: No NaN or Inf values found in weights
num_tensors_checked: 150
🟢 [PASS] weight_magnitudes: All weight magnitudes in reasonable ranges
min_threshold: 1e-06
max_threshold: 1000.0
🟡 [PASS] layer_norm_folding: Partial LN folding: ln2 (mean=1.000000) folded; ln1 (mean=0.235300) preserved (expected for combined QKV models)
folded: ['ln2']
not_folded: ['ln1']
🟢 [PASS] attention_output_centering: Attention output centering verified (worst_mean=0.000390) (6/16 blocks have attention)
mean: 0.00039015396032482386
tolerance: 0.01
n_attn_blocks: 6
🟢 [PASS] mlp_output_centering: MLP output centering verified (mean=0.000185)
mean: 0.00018529227236285806
tolerance: 0.01
🟢 [PASS] unembed_centering: Unembed centering verified (mean=0.000000)
mean: 8.756500524675914e-10
tolerance: 0.01
key: unembed.weight
🟢 [PASS] value_bias_folding: Skipped for GQA models (n_key_value_heads != n_heads)
is_gqa: True
n_heads: 32
n_kv_heads: 8
⏭️ weight_processing and weight_sharing skipped (no HT reference)
⏭️ [SKIPPED] weight_processing: Skipped (HookedTransformer not available for this model)
⏭️ [SKIPPED] weight_sharing: Skipped (HookedTransformer not available for this model)
🟢 [PASS] weight_modification: Weight modification propagates correctly (change: 0.013447)
change: 0.013446807861328125
2. Model Equivalence Benchmarks (Forward Pass)
Using saved Phase 1 bridge reference for equivalence comparison
🟢 [PASS] logits_equivalence: Tensors match within tolerance
atol: 0.01
rtol: 0.0001
🟢 [PASS] loss_equivalence: Scalars match: 2.597674 ≈ 2.597675
diff: 2.384185791015625e-07
atol: 0.001
3. Hook Registration Benchmarks
🟢 [PASS] hook_registry: Bridge has 787 registered hooks
hook_count: 787
4. Forward Hook Functionality Benchmarks
🟡 [PASS] hook_functionality: Ablation had minimal effect: 0.000000
effect: 0.0
🟢 [PASS] critical_forward_hooks: Bridge captured 8/14 critical hooks
captured: 8
expected: 14
🟡 [PASS] forward_hooks: 126/787 hooks didn't fire during forward pass
captured: 661
registered: 787
didnt_fire: ['blocks.1.hook_mlp_in', 'blocks.14.mlp.out.hook_out', 'blocks.5.hook_attn_in', 'blocks.5.attn.hook_attn_in', 'blocks.11.hook_mlp_in', 'blocks.5.hook_mlp_in', 'blocks.8.attn.hook_attn_in', 'blocks.8.hook_mlp_in', 'blocks.4.mlp.in.hook_in', 'blocks.10.attn.hook_result']
🟢 [PASS] gated_hooks_fire: All gated hooks fired on their supporting layers (54 activations across 5 hook stems, 3 flags tested)
fired_counts: {'hook_result': 6, 'hook_q_input': 12, 'hook_k_input': 12, 'hook_v_input': 12, 'hook_attn_in': 12}
tested_flags: ['use_attn_result', 'use_split_qkv_input', 'use_attn_in']
skipped: []
5. Activation Cache Benchmarks
🟢 [PASS] run_with_cache: run_with_cache successful with 661 cached activations
cache_size: 661
🟢 [PASS] activation_cache: Activation cache created with 661 entries
cache_size: 661
6. Backward Gradient Benchmarks
🟢 [PASS] gradient_computation: Gradients computed successfully
🟢 [PASS] critical_backward_hooks: Bridge captured 7/12 critical backward gradients
captured: 7
expected: 12
🟢 [PASS] backward_hooks: Bridge captured 555 backward hook gradients
gradient_count: 555
Cleaning up TransformerBridge (processed)...
================================================================================
BENCHMARK SUMMARY
================================================================================
Phase 1: 2 tests run
Passed: 2/2 (100.0%)
Failed: 0/2 (0.0%)
Phase 2: 13 tests run
Passed: 13/13 (100.0%)
Failed: 0/13 (0.0%)
Skipped: 2
Phase 3: 20 tests run
Passed: 20/20 (100.0%)
Failed: 0/20 (0.0%)
Skipped: 2
Phase 4: 1 tests run
Passed: 1/1 (100.0%)
Failed: 0/1 (0.0%)
Overall:
Total: 40 tests
Run: 36 tests
Skipped: 4 tests
Passed: 36/36 (100.0%)
Failed: 0/36 (0.0%)
================================================================================
VERIFIED: P1=100.0%, P2=100.0%, P3=100.0%, P4=99.3%, P7=None%, P8=None%
======================================================================
Verification Summary
======================================================================
Total tested: 1
Verified: 1
Skipped: 0
Failed: 0
Verified models:
- LiquidAI/LFM2-1.2B
Total time: 149.0s
Checkpoint cleared (run complete)
adil@Syeds-MacBook-Pro TransformerLens %
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds support for LFM2. Includes:
Verify model test ran on [LiquidAI/LFM2.5-230M, LiquidAI/LFM2-350M, LiquidAI/LFM2-1.2B]. Tests pass except harness fails on conv blocks due to difference in shape (channels last vs first). This could be fixed by adding transpose step for conv layers in test harness.
Also, compatibility mode test fails. I will paste test results for 1.2B model in comment below for reference.
Questions:
Fixes #1461
Type of change