mlsanityData to 7 Oct 2026

Spectral Archaeology: The Causal Topology of Model Evolution

arXiv 2601.03424 ↗First posted 6 Jan 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Behavioral benchmarks tell us textitwhat a model does, but not textithow. We introduce a training-free mechanistic probe using attention-graph spectra. Treating each layer as a token graph, we compute algebraic connectivity (λ2), smoothness, and spectral entropy. Across 12 models and 10 languages, these measures yield stable ``spectral fingerprints'' that expose discontinuities missed by standard evaluation. We report four results. (1) Models undergoing specific curriculum transitions (e.g., code-to-chat) show an English-only, syntax-triggered connectivity failure on non-canonical constructions, reaching Deltaλ2 ≈ -0.76. We term this scar textitPassive-Triggered Connectivity Collapse (PTCC). Analysis of the Phi lineage reveals that PTCC appears and resolves across developmental stages, implicating brittle curriculum shifts rather than synthetic data per se. (2) PTCC reflects a specialization trade-off: strengthened formal routing at the expense of stylistic flexibility. (3) We identify four recurrent processing strategies; simple frozen-threshold rules enable perfect forensic identification across lineages. (4) Mechanistically, PTCC localizes to a sparse Layer 2 ``compensatory patch'' of heads that fails under syntactic stress; activation steering can partially restore connectivity, recovering ≈ 38% of lost information flow. Finally, dominant topological regimes track tokenization density more than language identity, suggesting ``healthy'' geometry varies systematically across scripts. Overall, attention-graph spectra provide a practical tool for auditing and training-regime verification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

35 of 45 references.

References · 35

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2020Quantifying Attention Flow in Transformerscited by 759
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  11. 2023Textbooks Are All You Needcited by 561
  12. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
23 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2023The False Promise of Imitating Proprietary LLMscited by 254
  3. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  4. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  5. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  6. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  7. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  11. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  12. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  13. 2022Probing for the Usage of Grammatical Numbercited by 57
  14. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  15. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  16. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  17. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 42
  18. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  19. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  20. 2025EigenTrack: Spectral Activation Feature Tracking for Hallucination and Out-of-Distribution Detection in LLMs and VLMscited by 10
  21. 2025Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Modelscited by 3
  22. 2026Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoningcited by 3
  23. 2025Training-Free Spectral Fingerprints of Voice Processing in Transformerscited by 2

Built on by

Not cited yet.