# ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

arXiv: https://arxiv.org/abs/2607.05185 · HTML: https://mlsanity.com/paper/2607.05185/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [Understanding Reasoning in Thinking Language Models via Steering Vectors](https://mlsanity.com/paper/2506.18167/) (2025)
- [Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision](https://mlsanity.com/paper/2604.12002/) (2026)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration](https://mlsanity.com/paper/2601.18779/) (2026)
- [HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation](https://mlsanity.com/paper/2603.23871/) (2026)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Forking Paths in Neural Text Generation](https://mlsanity.com/paper/2412.07961/) (2024)
- [Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning](https://mlsanity.com/paper/2502.06533/) (2025)
- [Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes](https://mlsanity.com/paper/2601.18795/) (2026)

## Abstract (arXiv)

Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.