Data to 7 Oct 2026

Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training

arXiv 2311.09198 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) are equipped with longer text input capabilities than before, they are struggling to seek correct information in long contexts. The "lost in the middle" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA). Following these tasks, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. We release our model, Ziya-Reader to promote related research in the community.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

42 of 55 references.

References · 42

2017Attention Is All You Needcited by 59,932
2022Training language models to follow instructions with human feedbackcited by 15,712
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830

30 more

2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
2022LaMDA: Language Models for Dialog Applicationscited by 1,218
2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
2023Do Language Models Know When They're Hallucinating References?cited by 115
2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
2022Your Transformer May Not be as Powerful as You Expectcited by 55
2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
2022Humans decompose tasks by trading off utility and computational costcited by 18
2021Towards a Better Understanding Human Reading Comprehension with Brain Signalscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 6.

31 papers.

Sort:

By area:

2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
2025Shifting Long-Context LLMs Research from Input to Outputcited by 13
2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
2025Unstructured Evidence Attribution for Long Context Query Focused Summarizationcited by 5
2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
2026Intelligence Degradation in Long-Context LLMs: Critical Threshold Determination via Natural Length Distribution Analysiscited by 4
2025Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigationcited by 3
2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3

19 more

2024Mitigate Position Bias in Large Language Models via Scaling a Single Dimensioncited by 2
2024SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machinecited by 2
2025SEAL: Scaling to Emphasize Attention for Long-Context Retrievalcited by 2
2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
2024DoubleDipper: Improving Long-Context LLMs via Context Recyclingcited by 1
2024Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Modelscited by 1
2025END: Early Noise Dropping for Efficient and Effective Context Denoisingcited by 1
2025Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modelingcited by 1
2025Binary Autoencoder for Mechanistic Interpretability of Large Language Modelscited by 1
2025Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processingcited by 1
2024Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilitiescited by 0
2024Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Taskscited by 0
2025Parallel Key-Value Cache Fusion for Position Invariant RAGcited by 0
2025Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMscited by 0
2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
2026Improving Answer Extraction in Context-based Question Answering Systems Using LLMscited by 0
2026Unified KV Pooling to Accelerate Long-Context LLM Servingcited by 0
2026CRAFT: Learn the Schema, Execute the Plancited by 0
2026The Sirens' Song: When Proximal Background Context Overshadows Distant Evidencecited by 0