mlsanityData to 7 Oct 2026

LR^2Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems

arXiv 2502.17848 ↗First posted 25 Feb 2025cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent progress in Large Reasoning Models (LRMs) has significantly enhanced the reasoning abilities of Large Language Models (LLMs), empowering them to tackle increasingly complex tasks through reflection capabilities, such as making assumptions, backtracking, and self-refinement. However, effectively evaluating such reflection capabilities remains challenging due to the lack of appropriate benchmarks. To bridge this gap, we introduce LR2Bench, a novel benchmark designed to evaluate the Long-chain Reflective Reasoning capabilities of LLMs. LR2Bench comprises 850 samples across six Constraint Satisfaction Problems (CSPs) where reflective reasoning is crucial for deriving solutions that meet all given constraints. Each type of task focuses on distinct constraint patterns, such as knowledge-based, logical, and spatial constraints, providing a comprehensive evaluation of diverse problem-solving scenarios. Our extensive evaluation on both conventional LLMs and LRMs reveals that even the most advanced LRMs, such as DeepSeek-R1 and OpenAI o1-preview, struggle with tasks in LR2Bench, achieving an average Exact Match score of only 20.0% and 23.6%, respectively. These findings underscore the significant room for improvement in the reflective reasoning capabilities of current LLMs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 43 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

16 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey17 Feb

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2023Mistral 7Bcited by 3,065
  5. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  6. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  7. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  8. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  9. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  10. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  11. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  12. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizabilitycited by 18
3 more
  1. 2024FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?cited by 16
  2. 2024Reflection-Bench: Evaluating Epistemic Agency in Large Language Modelscited by 7
  3. 2024The Critique of Critiquecited by 5

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Modelscited by 16
  2. 2025CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generationcited by 9
  3. 2025Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspectivecited by 4
  4. 2025Towards Efficient Multimodal Unified Reasoning Model via Model Mergingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoning
  3. 2025Efficient Inference for Large Reasoning Models: A Survey
  4. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  5. 2026How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
  6. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  7. 2025Efficient Reasoning for Large Reasoning Language Models via Certainty-Guided Reflection Suppression
  8. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion