mlsanityData to 14 Oct 2026

The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

arXiv 2606.11198 ↗First posted 21 Apr 2026cs.CL cs.AIcited by 1

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution. We identify and formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples, due to their relational delimiters and repeated slot patterns, capture 2-3x more attention per token than semantically equivalent natural-language text (hato(KG) ≈ 0.70 vs. hato(neutral) ≈ 0.25), compressing demonstration attention by up to 42% -- regardless of whether the triples are relevant or noise. We develop a formal framework decomposing attention scores into semantic and structural components (Eq. 2), derive a compression bound (Proposition 1) connecting token-level format bias to demonstration attention loss, and show that the structural term governs how much attention is diverted while the semantic term governs whether this helps or hurts. This decoupling reveals two orthogonal axes for improving retrieval-augmented ICL: optimising retrieval quality (semantic axis) and reducing format-driven attention capture (structural axis). Empirically, across two model families (Mistral-7B, LLaMA-3-8B) and three QA benchmarks, we observe that source-task alignment dominates: task-matched BM25 retrieval achieves 58-62% on HotpotQA vs. ConceptNet's 25-27%, a >30 pp gap that dwarfs all gating strategies (≤2 pp). We derive five structure-aware mitigation strategies from the framework, ranging from zero-cost prompt modifications to training-time regularisation; format flattening (S3) is validated by both accuracy and attention-level evidence from a verbalized-triple control, while structural dispersal (S1) yields mixed results that illuminate the challenges of format-level intervention.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 23 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

21 of 23 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023Mistral 7Bcited by 3,164
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2023Active Retrieval Augmented Generationcited by 516
  11. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  12. 2023PRODIGY: Enabling In-context Learning Over Graphscited by 98
9 more
  1. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  2. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  3. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  4. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 44
  5. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  6. 2023Towards A Unified View of Answer Calibration for Multi-Step Reasoningcited by 8
  7. 2023The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learningcited by 7
  8. 2024"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"cited by 6
  9. 2025Schema for In-Context Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
  2. 2026Retrieval from Within: An Intrinsic Capability of Attention-Based Models
  3. 2024Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
  4. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
  5. 2023Attention Sorting Combats Recency Bias In Long Context Language Models
  6. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  7. 2024Quantifying reliance on external information over parametric knowledge during Retrieval Augmented Generation (RAG) using mechanistic analysis
  8. 2025Beyond Chunks and Graphs: Retrieval-Augmented Generation through Triplet-Driven Thinking