# PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

arXiv: https://arxiv.org/abs/2508.15180 · HTML: https://mlsanity.com/paper/2508.15180/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 23 of 23)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [LLMaAA: Making Large Language Models as Active Annotators](https://mlsanity.com/paper/2310.19596/) (2023)
- [MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs](https://mlsanity.com/paper/2402.16352/) (2024)
- [SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond](https://mlsanity.com/paper/2505.19641/) (2025)
- [AI-Assisted Generation of Difficult Math Questions](https://mlsanity.com/paper/2407.21009/) (2024)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving](https://mlsanity.com/paper/2502.20238/) (2025)
- [A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Data](https://mlsanity.com/paper/2109.03784/) (2021)
- [Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudoku](https://mlsanity.com/paper/2505.15993/) (2025)
- [Logic-of-Thought: Empowering Large Language Models with Logic Programs for Solving Puzzles in Natural Language](https://mlsanity.com/paper/2505.16114/) (2025)
- [LlamBERT: Large-scale low-cost data annotation in NLP](https://mlsanity.com/paper/2403.15938/) (2024)
- [Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs](https://mlsanity.com/paper/2409.10702/) (2024)

## Abstract (arXiv)

High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language models (LLMs). While recent data augmentation techniques have facilitated the creation of large-scale benchmarks, existing LLM-generated datasets often suffer from limited reliability, diversity, and scalability. To address these challenges, we introduce PuzzleClone, a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. Our approach features three key innovations: (1) encoding seed puzzles into structured logical specifications, (2) generating scalable variants through systematic variable and constraint randomization, and (3) ensuring validity via a reproduction mechanism. Applying PuzzleClone, we construct PC-83K, a benchmark comprising over 83K diverse and programmatically validated puzzles. The generated puzzles span a wide spectrum of difficulty and formats, posing significant challenges to current state-of-the-art models. Experimental results show that post training (SFT and RL) on PC-83K yields substantial improvements not only on the testset but also on various logic and mathematical benchmarks. Post training raises average performance on PC-83K from 14.5 to 66.0 and delivers consistent improvements across 7 logic and mathematical benchmarks up to 18.4 absolute percentage points (SATBench from 51.6 to 70.0). Our code and data are available at https://github.com/HiThink-Research/PuzzleClone.