mlsanityData to 13 Oct 2026

Anatomy of an Idiom: Tracing Non-Compositionality in Language Models

arXiv 2511.16467 ↗First posted 20 Nov 2025cs.CL cs.AI cs.LGcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

We investigate the processing of idiomatic expressions in transformer-based language models using a novel set of techniques for circuit discovery and analysis. First discovering circuits via a modified path patching algorithm, we find that idiom processing exhibits distinct computational patterns. We identify and investigate ``Idiom Heads,'' attention heads that frequently activate across different idioms, as well as enhanced attention between idiom tokens due to earlier processing, which we term ``augmented reception.'' We analyze these phenomena and the general features of the discovered circuits as mechanisms by which transformers balance computational efficiency and robustness. Finally, these findings provide insights into how transformers handle non-compositional language and suggest pathways for understanding the processing of more complex grammatical constructions.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 17 of its 27 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

17 of 27 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Open Problems in Mechanistic Interpretability27 Jan
  2. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models2 Jul
  3. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition1 Jul
  4. 2024Transcoders Find Interpretable LLM Feature Circuits17 Jun
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 11

  1. 2017Attention Is All You Needcited by 60,106
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,968
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,391
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 774
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 345
  7. 2023Localizing Model Behavior with Path Patchingcited by 188
  8. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 107
  9. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  10. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  11. 2024Semantics of Multiword Expressions in Transformer-Based Models: A Surveycited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models
  2. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models
  3. 2026All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs
  4. 2025Three tiers of computation in transformers and in brain architectures
  5. 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
  6. 2023Faith and Fate: Limits of Transformers on Compositionality
  7. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning
  8. 2026Where Animacy Lives in Large Language Models: Tracing the Circuits of the Animacy Concept