mlsanityData to 14 Oct 2026

Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation

arXiv 2503.06868 ↗First posted 10 Mar 2025cs.CL cs.AIcited by 3

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Existing long-text generation methods produce lengthy outputs from short inputs, leaving long-input-to-long-output generation underexplored. As input length increases, LLMs increasingly overlook information in the middle of the context--a limitation known as the "lost-in-the-middle" phenomenon--leading to inconsistent and incoherent outputs. To address this problem, we propose Retrieval-Augmented Long-Text Writer (RAL-Writer), a training-free framework consisting of a Planner that generates writing steps and a Writer that produces content according to these steps. The Writer jointly models semantic relevance and positional bias to compute importance scores, dynamically retrieve critical input segments, and strategically restate them. We also construct a benchmark dataset for long-input-to-long-output generation and introduce three evaluation metrics covering length, consistency, and quality. We evaluate RAL-Writer against comparable baselines on this dataset. The results demonstrate the effectiveness of our approach. Code is available at https://github.com/OnlyAR/RAL-Writer.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 27 of its 32 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

27 of 32 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  10. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  12. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
15 more
  1. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  2. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  3. 2024Make Your LLM Fully Utilize the Contextcited by 119
  4. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  5. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  6. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  7. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  8. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  9. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  10. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 32
  11. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  12. 2024PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Modelscited by 24
  13. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  14. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  15. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026TraceMem: Weaving Narrative Memory Schemata from User Conversational Tracescited by 2
  2. 2026SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?cited by 1
  3. 2025LLM as Attention-Informed NTM and Topic Modeling as long-input Generation: Interpretability and long-Context Capabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  2. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2024LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Models
  5. 2024Long Context RAG Performance of Large Language Models
  6. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  7. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
  8. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely