mlsanityData to 7 Oct 2026

The Dual-Route Model of Induction

arXiv 2504.03022 ↗First posted 3 Apr 2025cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Prior work on in-context copying has shown the existence of induction heads, which attend to and promote individual tokens during copying. In this work we discover a new type of induction head: concept-level induction heads, which copy entire lexical units instead of individual tokens. Concept induction heads learn to attend to the ends of multi-token words throughout training, working in parallel with token-level induction heads to copy meaningful text. We show that these heads are responsible for semantic tasks like word-level translation, whereas token induction heads are vital for tasks that can only be done verbatim (like copying nonsense tokens). These two "routes" operate independently: we show that ablation of token induction heads causes models to paraphrase where they would otherwise copy verbatim. By patching concept induction head outputs, we find that they contain language-independent word representations that mediate natural language translation, suggesting that LLMs represent abstract word meanings independent of language or form.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

36 of 53 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  9. 2017Word Translation Without Parallel Datacited by 480
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023Function Vectors in Large Language Modelscited by 292
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
24 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  2. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  3. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  4. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  5. 2019Adaptively Sparse Transformerscited by 154
  6. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  7. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  8. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  9. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  10. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  11. 2025Do Multilingual LLMs Think In English?cited by 75
  12. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  13. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  14. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  15. 2024Do language models plan ahead for future tokens?cited by 47
  16. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  17. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  18. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  19. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  20. 2024The broader spectrum of in-context learningcited by 33
  21. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  22. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  23. 2017Distinguishing Antonyms and Synonyms in a Pattern-based Neural Networkcited by 19
  24. 2024Token Erasure as a Footprint of Implicit Vocabulary Items in LLMscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 18
  2. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  3. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8
  4. 2025LLMs Process Lists With General Filter Headscited by 7
  5. 2025Mechanistic Indicators of Understanding in Large Language Modelscited by 6
  6. 2025Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Modelscited by 5
  7. 2026When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Trainingcited by 5
  8. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 4
  9. 2025In-Context Algebracited by 4
  10. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  11. 2025Vector Arithmetic in Concept and Token Subspacescited by 3
  12. 2026Pitfalls in Evaluating Interpretability Agentscited by 2
10 more
  1. 2025A circuit for predicting hierarchical structure in-context in Large Language Modelscited by 1
  2. 2026Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Modelscited by 1
  3. 2026Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretrainingcited by 1
  4. 2025In-Context Learning Without Copyingcited by 0
  5. 2026Emergent Structured Representations Support Flexible In-Context Inference in Large Language Modelscited by 0
  6. 2026France or Spain or Germany or France: A Neural Account of Non-Redundant Redundant Disjunctionscited by 0
  7. 2026Inference Time Causal Probing in LLMscited by 0
  8. 2026A Mechanistic Understanding of Pronoun Fidelity in LLMscited by 0
  9. 2026Using OCR Heads to Verbalize Image Semanticscited by 0
  10. 2026Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Contextcited by 0