mlsanityData to 14 Oct 2026

Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles

arXiv 2505.19914 ↗First posted 26 May 2025cs.CL cs.AIcited by 56

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs), such as OpenAI's o1 and DeepSeek's R1, excel at advanced reasoning tasks like math and coding via Reinforcement Learning with Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans without domain knowledge. We introduce Enigmata, the first comprehensive suite tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks across seven categories, each with 1) a generator that produces unlimited examples with controllable difficulty and 2) a rule-based verifier for automatic evaluation. This generator-verifier design supports scalable, multi-task RL training, fine-grained analysis, and seamless RLVR integration. We further propose Enigmata-Eval, a rigorous benchmark, and develop optimized multi-task RLVR strategies. Our trained model, Qwen2.5-32B-Enigmata, consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks like Enigmata-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes well to out-of-domain puzzle benchmarks and mathematical reasoning, with little multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking (20B activated parameters and 200B total parameters), puzzle data from Enigmata further boosts SoTA performance on advanced math and STEM reasoning tasks such as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization benefits of Enigmata. This work offers a unified, controllable framework for advancing logical reasoning in LLMs. Resources of this work can be found at https://seed-enigmata.github.io.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 45 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

31 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning10 Apr
  2. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond13 Mar
  3. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models3 Feb
  4. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning3 Feb
  5. 2023Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning13 Oct
  6. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  12. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
13 more
  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  4. 2019On the Measure of Intelligencecited by 144
  5. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  6. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  7. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
  8. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 70
  9. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 51
  10. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  11. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solvingcited by 11
  12. 2021A Puzzle-Based Dataset for Natural Language Inferencecited by 4
  13. 2024Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 33.

56 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  3. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  4. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  5. 2025Language Models that Think, Chat Bettercited by 24
  6. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  7. 2025Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Modelscited by 11
  8. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problemscited by 11
  9. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  10. 2025ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMscited by 9
  11. 2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
  12. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusionscited by 6
38 more
  1. 2025Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoningcited by 6
  2. 2025Think Visually, Reason Textually: Vision-Language Synergy in ARCcited by 6
  3. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 5
  4. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  5. 2026Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoningcited by 5
  6. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Successcited by 4
  7. 2025MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimizationcited by 4
  8. 2025Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Modelscited by 4
  9. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  10. 2025DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Modelcited by 3
  11. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMscited by 3
  12. 2026TopoBench: Benchmarking LLMs on Hard Topological Reasoningcited by 3
  13. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 3
  14. 2025Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compressioncited by 2
  15. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  16. 2025SPHINX: A Synthetic Environment for Visual Perception and Reasoningcited by 2
  17. 2026The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Modelscited by 2
  18. 2026Group Pattern Selection Optimization: Let LRMs Pick the Right Pattern for Reasoningcited by 1
  19. 2026TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimizationcited by 1
  20. 2026Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Modelscited by 1
  21. 2026Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Keycited by 1
  22. 2026ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimizationcited by 1
  23. 2026Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoningcited by 1
  24. 2026Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Promptscited by 1
  25. 2025Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learningcited by 0
  26. 2025NSTR: Neural Spectral Transport Representation for Space-Varying Frequency Fieldscited by 0
  27. 2025Scaling Implicit Fields via Hypernetwork-Driven Multiscale Coordinate Transformationscited by 0
  28. 2026LsrIF: Enhancing Logic-Structured Instruction Following of Large Language Modelscited by 0
  29. 2026GenPlanner: From Noise to Plans -- Emergent Reasoning in Flow Matching and Diffusion Modelscited by 0
  30. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  31. 2026MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMscited by 0
  32. 2026Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMscited by 0
  33. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spacescited by 0
  34. 2026Scaling of Capability and Efficiency at Inference Time in Large Reasoning Modelscited by 0
  35. 2026Rufus-Air: An Open LLM Post-Training Recipecited by 0
  36. 2026Self-Play Search Distillation for Large Language Model Reasoningcited by 0
  37. 2026Witness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environmentscited by 0
  38. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
  2. 2025PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey
  4. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
  5. 2025Reinforcing General Reasoning without Verifiers
  6. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
  7. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  8. 2025RiddleBench: A New Generative Reasoning Benchmark for LLMs