PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language models (LLMs). While recent data augmentation techniques have facilitated the creation of large-scale benchmarks, existing LLM-generated datasets often suffer from limited reliability, diversity, and scalability. To address these challenges, we introduce PuzzleClone, a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. Our approach features three key innovations: (1) encoding seed puzzles into structured logical specifications, (2) generating scalable variants through systematic variable and constraint randomization, and (3) ensuring validity via a reproduction mechanism. Applying PuzzleClone, we construct PC-83K, a benchmark comprising over 83K diverse and programmatically validated puzzles. The generated puzzles span a wide spectrum of difficulty and formats, posing significant challenges to current state-of-the-art models. Experimental results show that post training (SFT and RL) on PC-83K yields substantial improvements not only on the testset but also on various logic and mathematical benchmarks. Post training raises average performance on PC-83K from 14.5 to 66.0 and delivers consistent improvements across 7 logic and mathematical benchmarks up to 18.4 absolute percentage points (SATBench from 51.6 to 70.0). Our code and data are available at https://github.com/HiThink-Research/PuzzleClone.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 53 references that are in the map. First posted on arXiv 21 Aug 2025.
Builds on
23 of 53 references.
Closest ancestors · 4
Same direction, newest first
- 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving27 Feb
- 2024AI-Assisted Generation of Difficult Math Questions30 Jul
- 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs26 Feb
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
Other references · 19
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
- 2021A Survey of Data Augmentation Approaches for NLPcited by 409
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
- 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
- 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 120
- 2021A Survey on Data Augmentation for Text Classificationcited by 111
- 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
- 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
7 more
- 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 50
- 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 33
- 2021A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Datacited by 6
- 2025Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudokucited by 5
- 2025Logic-of-Thought: Empowering Large Language Models with Logic Programs for Solving Puzzles in Natural Languagecited by 5
- 2024LlamBERT: Large-scale low-cost data annotation in NLPcited by 3
- 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
- 2026MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs
- 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
- 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
- 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
- 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- 2026ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation
- 2025Saturation-Driven Dataset Generation for LLM Mathematical Reasoning in the TPTP Ecosystem