mlsanityData to 7 Oct 2026

Make Your LLM Fully Utilize the Context

arXiv 2404.16811 ↗First posted 25 Apr 2024cs.CL cs.AIcited by 118

Not on the map. Why?

Abstract (arXiv)

While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents information-intensive (IN2) training, a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this information-intensive training on Mistral-7B, we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

56 of 67 references.

References · 56

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2023Mistral 7Bcited by 3,065
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  13. 2023StarCoder: may the source be with you!cited by 790
  14. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  15. 2024Yi: Open Foundation Models by 01.AIcited by 706
  16. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  17. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  18. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  19. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  20. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  21. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  22. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  23. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  24. 2024InternLM2 Technical Reportcited by 391
  25. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  26. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  27. 2021Efficient Attentions for Long Document Summarizationcited by 342
  28. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  29. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  30. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  31. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  32. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  33. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  34. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  35. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  36. 2023Retrieval meets Long Context Large Language Modelscited by 114
  37. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  38. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 64; 2026: 26.

118 papers.

Sort:

By area:

  1. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  2. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  4. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  5. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  6. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  7. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  8. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  9. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  10. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  11. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  12. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
38 more
  1. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  2. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  3. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  4. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  5. 2024LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Modelscited by 17
  6. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  7. 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14
  8. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  9. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 13
  10. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  11. 2024AI-native Memory: A Pathway from LLMs Towards AGIcited by 12
  12. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  13. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  14. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  15. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  16. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  17. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  18. 2024On the Loss of Context-awareness in General Instruction Fine-tuningcited by 10
  19. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  20. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  21. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  22. 2024Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Taskscited by 8
  23. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  24. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?cited by 7
  25. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 7
  26. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  27. 2024Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMscited by 6
  28. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  29. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  30. 2024Order-Independence Without Fine Tuningcited by 5
  31. 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
  32. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  33. 2025DocMEdit: Towards Document-Level Model Editingcited by 5
  34. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  35. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  36. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  37. 2026Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industrycited by 5
  38. 2026Large Language Model Agents Are Not Always Faithful Self-Evolverscited by 5

50 most cited of 118 All 118 (JSON)