mlsanityData to 14 Oct 2026

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

arXiv 2604.14140 ↗First posted 15 Apr 2026cs.LG cs.AIcited by 6

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 38 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

31 of 38 references.

References · 31

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  6. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  7. 2025Kimi K2: Open Agentic Intelligencecited by 567
  8. 2025Humanity's Last Examcited by 491
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  11. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  12. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
19 more
  1. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  2. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 133
  3. 2025Recursive Language Modelscited by 104
  4. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  5. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  6. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  7. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  8. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 30
  9. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  10. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  11. 2025UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarioscited by 26
  12. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 24
  13. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 17
  14. 2025Assessing the Chemical Intelligence of Large Language Modelscited by 17
  15. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 13
  16. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  17. 2025Generating Creative Chess Puzzlescited by 8
  18. 2026WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoningcited by 8
  19. 2025Technical Report: Evaluating Goal Drift in Language Model Agentscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026SentinelBench: A Benchmark for Long-Running Monitoring Agentscited by 3
  2. 2026Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seekingcited by 2
  3. 2026Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?cited by 1
  4. 2026Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languagescited by 1
  5. 2026SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scalingcited by 0
  6. 2026Code Understanding is a Bottleneck for Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
  2. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  3. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
  4. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  5. 2025Early Stopping Chain-of-thoughts in Large Language Models
  6. 2025Long Chain-of-Thought Reasoning Across Languages
  7. 2025The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think
  8. 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models