mlsanityData to 14 Oct 2026

AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models

arXiv 2502.16906 ↗First posted 24 Feb 2025cs.CLcited by 31

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While logical reasoning evaluation of Large Language Models (LLMs) has attracted significant attention, existing benchmarks predominantly rely on multiple-choice formats that are vulnerable to random guessing, leading to overestimated performance and substantial performance fluctuations. To obtain more accurate assessments of models' reasoning capabilities, we propose an automated method for synthesizing open-ended logic puzzles, and use it to develop a bilingual benchmark, AutoLogi. Our approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities. Extensive evaluation of eight modern LLMs shows that AutoLogi can better reflect true model capabilities, with performance scores spanning from 35% to 73% compared to the narrower range of 21% to 37% on the source multiple-choice dataset. Beyond benchmark creation, this synthesis method can generate high-quality training data by incorporating program verifiers into the rejection sampling process, enabling systematic enhancement of LLMs' reasoning capabilities across diverse datasets.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 31 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

22 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning24 Oct
  2. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks23 Oct
  3. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct
  4. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models19 Apr
  5. 2022PAL: Program-aided Language Models18 Nov
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  7. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  9. 2020Transformers as Soft Reasoners over Languagecited by 270
  10. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  11. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  12. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 69
3 more
  1. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  2. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  3. 2023Few-shot Unified Question Answering: Tuning Models or Prompts?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 22.

31 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  5. 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 33
  6. 2026DOPD: Dual On-policy Distillationcited by 12
  7. 2026Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimizationcited by 9
  8. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  9. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  10. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RLcited by 5
  11. 2025Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoningcited by 5
  12. 2026Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environmentscited by 5
19 more
  1. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  2. 2025CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Densitycited by 3
  3. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifierscited by 2
  4. 2026What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillationcited by 2
  5. 2026Evaluating Robustness of Reasoning Models on Parameterized Logical Problemscited by 1
  6. 2026Scaling the Scaling Logic: Agentic Meta-Synthesis of Logic Reasoningcited by 1
  7. 2026TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordinationcited by 1
  8. 2025From Language to Logic: A Bi-Level Framework for Structured Reasoningcited by 0
  9. 2026VeRA: Renewing Reasoning Benchmarks with Executable Specificationscited by 0
  10. 2026Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMscited by 0
  11. 2026LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardeningcited by 0
  12. 2026JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Datacited by 0
  13. 2026State commitment learning: training language models to distinguish computation from memorycited by 0
  14. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spacescited by 0
  15. 2026DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordinationcited by 0
  16. 2026NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Researchcited by 0
  17. 2026NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gapcited by 0
  18. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Modelscited by 0
  19. 2026SAT-RL: Training Language Models for Logical Reasoning with Symbolic Solverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  4. 2026QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
  5. 2024LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
  6. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  7. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  8. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus