mlsanityData to 7 Oct 2026

Attention Sorting Combats Recency Bias In Long Context Language Models

arXiv 2310.01427 ↗First posted 28 Sep 2023cs.CL cs.AIcited by 89

Not on the map. Why?

Abstract (arXiv)

Current language models often fail to incorporate long contexts efficiently during generation. We show that a major contributor to this issue are attention priors that are likely learned during pre-training: relevant information located earlier in context is attended to less on average. Yet even when models fail to use the information from a relevant document in their response, they still pay preferential attention to that document compared to an irrelevant document at the same position. We leverage this fact to introduce ``attention sorting'': perform one step of decoding, sort documents by the attention they receive (highest attention going last), repeat the process, generate the answer with the newly sorted context. We find that attention sorting improves performance of long context models. Our findings highlight some challenges in using off-the-shelf language models for retrieval augmented generation.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

32 of 37 references.

References · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
20 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2019Is Attention Interpretable?cited by 387
  13. 2023Larger language models do in-context learning differentlycited by 386
  14. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  15. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  16. 2022Memorizing Transformerscited by 219
  17. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  18. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  19. 2019Attention Interpretability Across NLP Taskscited by 71
  20. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 27; 2025: 36; 2026: 23.

89 papers.

Sort:

By area:

  1. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  2. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  3. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  4. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  5. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  6. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  7. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
  8. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  9. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 32
  10. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  11. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  12. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
38 more
  1. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  2. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  3. 2025SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidencecited by 18
  4. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  5. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 17
  6. 2025Towards Enforcing Company Policy Adherence in Agentic Workflowscited by 17
  7. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  8. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  9. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  10. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  11. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
  12. 2024AI-native Memory: A Pathway from LLMs Towards AGIcited by 12
  13. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  14. 2025Learning to Attribute with Attentioncited by 11
  15. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  16. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 8
  17. 2024Where is the answer? Investigating Positional Bias in Language Model Knowledge Extractioncited by 6
  18. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  19. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  20. 2025Quantifying Cognitive Bias Induction in LLM-Generated Contentcited by 6
  21. 2026FASA: Frequency-aware Sparse Attentioncited by 6
  22. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  23. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Textcited by 5
  24. 2023Training With "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-context Taskscited by 5
  25. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  26. 2025Exploiting Primacy Effect To Improve Large Language Modelscited by 5
  27. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  28. 2026ReAttn: Improving Attention-based Re-ranking via Attention Re-weightingcited by 5
  29. 2026Interpretability Can Be Actionablecited by 5
  30. 2025Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Trainingcited by 4
  31. 2025Stepwise Informativeness Search for Efficient and Effective LLM Reasoningcited by 4
  32. 2025How Good Are LLMs at Processing Tool Outputs?cited by 4
  33. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
  34. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  35. 2025VarDrop: Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecastingcited by 3
  36. 2025Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAGcited by 3
  37. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  38. 2025RePo: Language Models with Context Re-Positioningcited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
  2. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  3. 2026Retrieval from Within: An Intrinsic Capability of Attention-Based Models
  4. 2025Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
  5. 2025AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
  6. 2026The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
  7. 2026Language Models Can Control Their Own Attention
  8. 2025Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias