mlsanityData to 14 Oct 2026

LooGLE: Can Long-Context Language Models Understand Long Contexts?

arXiv 2311.04939 ↗First posted 8 Nov 2023cs.CL cs.AIcited by 202

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".

Key facts. Cited by 202 papers in the map (data through 14 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

43 of 50 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  10. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
31 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  3. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  4. 2021Efficient Attentions for Long Document Summarizationcited by 345
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  7. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  8. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  10. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  11. 2022Recurrent Memory Transformercited by 203
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  13. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  14. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  15. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  16. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  17. 2020Extractive Opinion Summarization in Quantized Transformer Spacescited by 76
  18. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  19. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  20. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  21. 2023Calibrating LLM-Based Evaluatorcited by 53
  22. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  23. 2023A Survey on Long Text Modeling with Transformerscited by 47
  24. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  25. 2023Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5cited by 42
  26. 2021Is Sparse Attention more Interpretable?cited by 31
  27. 2023Fine-tuning and aligning question answering models for complex information extraction taskscited by 5
  28. 2023Retrieval for Extremely Long Queries and Documents with RPRS: a Highly Efficient and Effective Transformer-based Re-Rankercited by 4
  29. 2023Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and Considerations in Real-World Scenarioscited by 4
  30. 2023Augmenting text for spoken language understanding with Large Language Modelscited by 2
  31. 2023BugMentor: Generating Answers to Follow-up Questions from Software Bug Reports using Structured Information Retrieval and Neural Text Generationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 70; 2025: 76; 2026: 55.

202 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  3. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  5. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  6. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  8. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  9. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  11. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
38 more
  1. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  2. 2024Make Your LLM Fully Utilize the Contextcited by 119
  3. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  4. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  5. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  7. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 97
  8. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  9. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  10. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  11. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 74
  12. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  13. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  14. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  15. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  16. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  17. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  18. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  19. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  20. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 41
  21. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  22. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  23. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 33
  24. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  25. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  26. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Modelscited by 29
  27. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  28. 2023D-Bot: Database Diagnosis System using Large Language Modelscited by 28
  29. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  30. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  31. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  32. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  33. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novelscited by 21
  34. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  35. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  36. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  37. 2026Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systemscited by 20
  38. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 19

50 most cited of 202 All 202 (JSON)