mlsanityData to 14 Oct 2026

The Two-Hump Problem: Bridging the Difficulty Gap in Mathematical Reinforcement Learning

arXiv 2606.21611 ↗First posted 19 Jun 2026cs.LG cs.AI math.GRcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Mathematical search problems present a unique challenge for Reinforcement Learning (RL) due to vast search spaces and sparse rewards. In previous works, the Andrews-Curtis (AC) conjecture was established as an illustrative example of such problems. In this work, we identify a critical structural barrier in the AC landscape: a "Two-Hump" distribution, where problem instances are either trivially solvable or effectively impossible, with a scarcity of intermediate "hard-but-solvable" instances required for effective learning. We tackle this challenge through two primary avenues: novel data generation techniques to populate the difficulty gap, and significant algorithmic enhancements including the introduction of supermoves and Transformer-based architectures. We demonstrate substantial performance improvements over previous baselines, and release new comprehensive benchmark datasets including AC-19 (125,192 AC-trivial presentations of varying difficulty with length at most 19) and AC-1M (1,136,154 hard AC-trivial presentations of length at most 30), the first large-scale, publicly available datasets of this kind.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 35 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

12 of 35 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  4. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  5. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  6. 2020First return, then explorecited by 266
  7. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  8. 2022Formal Mathematics Statement Curriculum Learningcited by 142
  9. 2019Learning to Prove Theorems via Interacting with Proof Assistantscited by 109
  10. 2022Discovered Policy Optimisationcited by 104
  11. 2020Learning to Unknotcited by 40
  12. 2024What makes math problems hard for reinforcement learning: a case studycited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models
  2. 2026Mathematical methods of reinforcement learning
  3. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  4. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
  5. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation
  6. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  7. 2024Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning
  8. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning