mlsanityData to 7 Oct 2026

A Comprehensive Survey on Long Context Language Modeling

arXiv 2503.17407 ↗First posted 20 Mar 2025cs.CL cs.LGcited by 113

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: hrefhttps://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modelingcolor[RGB]175,36,67LCLM-Horizon.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Mar 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 59; 2026: 54.

113 papers.

Sort:

By area:

  1. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  2. 2025A Survey on Latent Reasoningcited by 71
  3. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  4. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  5. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  6. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  7. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  8. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  9. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  10. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 24
  11. 2025Mid-Training of Large Language Models: A Surveycited by 18
  12. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
38 more
  1. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 15
  2. 2025CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalizationcited by 15
  3. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  4. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  5. 2025Unveiling the Potential of Diffusion Large Language Model in Controllable Generationcited by 14
  6. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  7. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  8. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agentscited by 13
  9. 2025KAT-V1: Kwai-AutoThink Technical Reportcited by 12
  10. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 12
  11. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  12. 2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?cited by 11
  13. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  14. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  15. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Modelscited by 10
  16. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  17. 2026Toward Generalist Autonomous Research via Hypothesis-Tree Refinementcited by 10
  18. 2025Structured Memory Mechanisms for Stable Context Representation in Large Language Modelscited by 9
  19. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  20. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  21. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 9
  22. 2026Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agentscited by 9
  23. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 8
  24. 2026Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Managementcited by 8
  25. 2026When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningcited by 8
  26. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  27. 2026Governed Shared Memory for Multi-Agent LLM Systemscited by 8
  28. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  29. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  30. 2026MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaborationcited by 7
  31. 2026BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMscited by 7
  32. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Trainingcited by 7
  33. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  34. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  35. 2025Do LLMs Truly Understand When a Precedent Is Overruled?cited by 6
  36. 2025Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Modelscited by 6
  37. 2026MetaMem: Evolving Meta-Memory for Knowledge Utilization through Self-Reflective Symbolic Optimizationcited by 6
  38. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memorycited by 6

50 most cited of 113 All 113 (JSON)