mlsanityData to 7 Oct 2026

Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads

arXiv 2602.22453 ↗First posted 25 Feb 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Retrieval heads, a subset of attention heads in Transformers, were studied in English, showing its crucial role in retrieving information from the context. We expand the study of retrieval heads to multilingual context and find that while nearly half of all retrieval heads are often shared across multiple languages, language-specific retrieval heads also emerge. We further design a cross-lingual needle-in-the-haystack task, to identify textitRetrieval Transition Heads (RTH) that retrieve key information and further map it to target-language output. Our experiments reveal that RTHs do not always overlap with RH, and can be more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuAD) and two model families (Qwen2.5-7B Instruct and Llama3.1-8B Instruct), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Specifically, for Llama3.1-8B Instruct, masking the top-25 RTHs results in an average 37.3-point drop (against 19.5-point drop from RH masking) in reasoning accuracy (MMLU-ProX, MGSM) and 9.0 F1-score reduction (against 5.1 drop for RH) in extractive QA (MLQA, XQuAD). Our work advances the understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

19 of 25 references.

References · 19

  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  5. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  7. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  8. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  9. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  10. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  11. 2025Do Multilingual LLMs Think In English?cited by 75
  12. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
7 more
  1. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  2. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  3. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  4. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  5. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  6. 2025Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimizationcited by 17
  7. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
  2. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  3. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
  4. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoning
  5. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation
  6. 2026Multilinguality in Hybrid Attention LLMs
  7. 2026The Multilingual Curse at the Retrieval Layer: Evidence from Amharic
  8. 2026Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads