ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
Not on the map. Why?
Abstract (arXiv)
Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
21 of 28 references.
References · 21
- 2015Distilling the Knowledge in a Neural Networkcited by 10,180
- 2020Knowledge Distillation: A Surveycited by 1,590
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 873
- 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 862
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 414
- 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 375
- 2026Reinforcement Learning via Self-Distillationcited by 310
- 2025Training Language Models to Reason Efficientlycited by 245
- 2026Self-Distillation Enables Continual Learningcited by 234
- 2026Self-Distilled RLVRcited by 121
- 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 98
- 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 81
9 more
- 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 71
- 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 64
- 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 54
- 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 41
- 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 40
- 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 39
- 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
- 2024Forking Paths in Neural Text Generationcited by 31
- 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios
- 2025HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games
- 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoning
- 2025PuzzleJAX: A Benchmark for Reasoning and Learning
- 2025MastermindEval: A Simple But Scalable Reasoning Benchmark
- 2025LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
- 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
- 2026AGEL-Comp: A Neuro-Symbolic Framework for Compositional Generalization in Interactive Agents