# AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models

arXiv: https://arxiv.org/abs/2502.16906 · HTML: https://mlsanity.com/paper/2502.16906/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning](https://mlsanity.com/paper/2310.16049/) (2023)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)
- [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://mlsanity.com/paper/2007.08124/) (2020)
- [AR-LSAT: Investigating Analytical Reasoning of Text](https://mlsanity.com/paper/2104.06598/) (2021)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [LeTI: Learning to Generate from Textual Interactions](https://mlsanity.com/paper/2305.10314/) (2023)
- [CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks](https://mlsanity.com/paper/2310.15239/) (2023)
- [Few-shot Unified Question Answering: Tuning Models or Prompts?](https://mlsanity.com/paper/2305.14569/) (2023)

## Built on by (most cited first, 30 of 31)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [DOPD: Dual On-policy Distillation](https://mlsanity.com/paper/2606.30626/) (2026)
- [Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization](https://mlsanity.com/paper/2602.02188/) (2026)
- [WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models](https://mlsanity.com/paper/2602.12135/) (2026)
- [Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale](https://mlsanity.com/paper/2606.15079/) (2026)
- [Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL](https://mlsanity.com/paper/2509.06024/) (2025)
- [Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning](https://mlsanity.com/paper/2509.18083/) (2025)
- [Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environments](https://mlsanity.com/paper/2603.10793/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density](https://mlsanity.com/paper/2509.18458/) (2025)
- [DARL: Encouraging Diverse Answers for General Reasoning without Verifiers](https://mlsanity.com/paper/2601.14700/) (2026)
- [What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.25623/) (2026)
- [Evaluating Robustness of Reasoning Models on Parameterized Logical Problems](https://mlsanity.com/paper/2602.12665/) (2026)
- [Scaling the Scaling Logic: Agentic Meta-Synthesis of Logic Reasoning](https://mlsanity.com/paper/2602.13218/) (2026)
- [TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination](https://mlsanity.com/paper/2605.15207/) (2026)
- [From Language to Logic: A Bi-Level Framework for Structured Reasoning](https://mlsanity.com/paper/2507.08501/) (2025)
- [VeRA: Renewing Reasoning Benchmarks with Executable Specifications](https://mlsanity.com/paper/2602.13217/) (2026)
- [Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs](https://mlsanity.com/paper/2605.07731/) (2026)
- [LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening](https://mlsanity.com/paper/2605.19597/) (2026)
- [JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data](https://mlsanity.com/paper/2605.24414/) (2026)
- [State commitment learning: training language models to distinguish computation from memory](https://mlsanity.com/paper/2606.05201/) (2026)
- [Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces](https://mlsanity.com/paper/2606.05464/) (2026)
- [DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination](https://mlsanity.com/paper/2606.08068/) (2026)
- [NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research](https://mlsanity.com/paper/2606.26671/) (2026)
- [NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap](https://mlsanity.com/paper/2608.04397/) (2026)
- [Reducing Pretraining-Generation Mismatch in Diffusion Language Models](https://mlsanity.com/paper/2608.09424/) (2026)

## Abstract (arXiv)

While logical reasoning evaluation of Large Language Models (LLMs) has attracted significant attention, existing benchmarks predominantly rely on multiple-choice formats that are vulnerable to random guessing, leading to overestimated performance and substantial performance fluctuations. To obtain more accurate assessments of models' reasoning capabilities, we propose an automated method for synthesizing open-ended logic puzzles, and use it to develop a bilingual benchmark, AutoLogi. Our approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities. Extensive evaluation of eight modern LLMs shows that AutoLogi can better reflect true model capabilities, with performance scores spanning from 35% to 73% compared to the narrower range of 21% to 37% on the source multiple-choice dataset. Beyond benchmark creation, this synthesis method can generate high-quality training data by incorporating program verifiers into the rejection sampling process, enabling systematic enhancement of LLMs' reasoning capabilities across diverse datasets.