mlsanityData to 7 Oct 2026

LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion

arXiv 2501.15089 ↗First posted 25 Jan 2025cs.CLcited by 47

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason under https://huggingface.co/datasets/lz1bytedance/LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 54 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

33 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools18 Jun
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Mistral 7Bcited by 3,065
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
19 more
  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  4. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  5. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  8. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  9. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  10. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  11. 2023Retrieval meets Long Context Large Language Modelscited by 114
  12. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  13. 2022Parallel Context Windows for Large Language Modelscited by 81
  14. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  15. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  16. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 43
  17. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  18. 2024E^2-LLM: Efficient and Extreme Length Extension of Large Language Modelscited by 11
  19. 2024LongFin: A Multimodal Document Understanding Model for Long Financial Domain Documentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 24.

47 papers.

Sort:

By area:

  1. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  2. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  3. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  4. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  5. 2025UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarioscited by 22
  6. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  7. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  8. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 13
  9. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 12
  10. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 12
  11. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  12. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
35 more
  1. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  2. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  3. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  4. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  5. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  6. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Modelscited by 3
  7. 2025CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Densitycited by 3
  8. 2025Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Modelscited by 3
  9. 2026ConvexBench: Can LLMs Recognize Convex Functions?cited by 3
  10. 2026Do Reasoning LLMs Refuse What They Infer in Long Contexts?cited by 3
  11. 2026Reasoning Shift: How Context Silently Shortens LLM Reasoningcited by 3
  12. 2025EssayBench: Evaluating Large Language Models in Multi-Genre Chinese Essay Writingcited by 2
  13. 2026SciMDR: Advancing Scientific Multimodal Document Reasoningcited by 2
  14. 2026DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contextscited by 2
  15. 2026Information-Aware KV Cache Compression for Long Reasoningcited by 2
  16. 2025Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysiscited by 1
  17. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  18. 2026LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Modelscited by 1
  19. 2026Probing How Scalable Table Data Enhances General Long-Context Reasoningcited by 1
  20. 2026A Decomposition Perspective to Long-context Reasoning for LLMscited by 1
  21. 2026Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarkscited by 1
  22. 2026LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewardscited by 1
  23. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  24. 2025Context Is Not Comprehensioncited by 0
  25. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novelscited by 0
  26. 2026PPA-Plan: Proactive Pitfall Avoidance for Reliable Planning in Long-Context LLM Reasoningcited by 0
  27. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environmentscited by 0
  28. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learningcited by 0
  29. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learningcited by 0
  30. 2026Evidence-State Rewards for Long-Context Reasoningcited by 0
  31. 2026ContractScrub: A benchmark for final review of legal contractscited by 0
  32. 2026DualSQL: Text-to-SQL with Multi-Agent Reinforcement Learningcited by 0
  33. 2026Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Tracescited by 0
  34. 2026StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learningcited by 0
  35. 2026SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  2. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  4. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  5. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  6. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models
  7. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models
  8. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries