mlsanityData to 7 Oct 2026

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

arXiv 2406.13121 ↗First posted 19 Jun 2024cs.CL cs.AI cs.IRcited by 53

Not on the map. Why?

Abstract (arXiv)

Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

57 of 69 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  13. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  14. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  15. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  16. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  17. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  18. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  19. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  20. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  21. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  22. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  23. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  24. 2020Autoregressive Entity Retrievalcited by 333
  25. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  26. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  27. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  28. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  29. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  30. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
  31. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 138
  32. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  33. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  34. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  35. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  36. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 95
  37. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  38. 2024Repetition Improves Language Model Embeddingscited by 70

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 24; 2026: 11.

53 papers.

Sort:

By area:

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  2. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  3. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  4. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  5. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  6. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  7. 2024Long Context RAG Performance of Large Language Modelscited by 23
  8. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  9. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  10. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  11. 2024MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answeringcited by 17
  12. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
38 more
  1. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  2. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  3. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  4. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  5. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  6. 2025LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searcherscited by 10
  7. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMscited by 9
  8. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  9. 2025Scalable In-context Ranking with Generative Modelscited by 8
  10. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  11. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problemscited by 7
  12. 2025From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactionscited by 7
  13. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  14. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  15. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  16. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Contextcited by 6
  17. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  18. 2025Efficient and Asymptotically Unbiased Constrained Decoding for Large Language Modelscited by 5
  19. 2025Scaling Textual Gradients via Sampling-Based Momentumcited by 5
  20. 2024Bootstrap Your Own Context Lengthcited by 4
  21. 2025PERK: Long-Context Reasoning as Test-Time Learningcited by 4
  22. 2025LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrievalcited by 4
  23. 2026Recursive Agent Harnessescited by 4
  24. 2024Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Taskscited by 3
  25. 2024How Does Knowledge Selection Help Retrieval Augmented Generation?cited by 3
  26. 2024PRISM: Efficient Long-Range Reasoning With Short-Context LLMscited by 3
  27. 2025CORG: Generating Answers from Complex, Interrelated Contextscited by 3
  28. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Modelscited by 2
  29. 2025Evaluating the Retrieval Robustness of Large Language Modelscited by 2
  30. 2025Retrieval Augmented Question Answering: When Should LLMs Admit Ignorance?cited by 2
  31. 2026ATLAS: All-round Testing of Long-context Abilities across Scalescited by 2
  32. 2026Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scalecited by 2
  33. 2025From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learningcited by 1
  34. 2026Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoderscited by 1
  35. 2026RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrievalcited by 1
  36. 2026LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexingcited by 1
  37. 2025Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QAcited by 0
  38. 2026Scalable Model-Based Clustering with Sequential Monte Carlocited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps
  2. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
  3. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL
  4. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Models
  5. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  6. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
  7. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  8. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely