# What makes math problems hard for reinforcement learning: a case study

arXiv: https://arxiv.org/abs/2408.15332 · HTML: https://mlsanity.com/paper/2408.15332/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics](https://mlsanity.com/paper/2302.12433/) (2023)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://mlsanity.com/paper/1611.01462/) (2016)
- [Lean-STaR: Learning to Interleave Thinking and Proving](https://mlsanity.com/paper/2407.10040/) (2024)
- [giotto-tda: A Topological Data Analysis Toolkit for Machine Learning and Data Exploration](https://mlsanity.com/paper/2004.02551/) (2020)
- [A2C is a special case of PPO](https://mlsanity.com/paper/2205.09123/) (2022)
- [Peano: Learning Formal Mathematical Reasoning](https://mlsanity.com/paper/2211.15864/) (2022)
- [The Simplex Tree: an Efficient Data Structure for General Simplicial Complexes](https://mlsanity.com/paper/2001.02581/) (2020)
- [Distance Metric Ensemble Learning and the Andrews-Curtis Conjecture](https://mlsanity.com/paper/1606.01412/) (2016)

## Built on by (most cited first, 9 of 9)

- [CayleyPy RL: Pathfinding and Reinforcement Learning on Cayley Graphs](https://mlsanity.com/paper/2502.18663/) (2025)
- [Flow-based Extremal Mathematical Structure Discovery](https://mlsanity.com/paper/2601.18005/) (2026)
- [Diffusion Models for Cayley Graphs](https://mlsanity.com/paper/2503.05558/) (2025)
- [CayleyPy Growth: Efficient growth computations and hundreds of new conjectures on Cayley graphs (Brief version)](https://mlsanity.com/paper/2509.19162/) (2025)
- [An AI enhanced approach to the tree unimodality conjecture](https://mlsanity.com/paper/2510.18826/) (2025)
- [Discovering mathematical concepts through a multi-agent system](https://mlsanity.com/paper/2603.04528/) (2026)
- [TriSearch: Learning to Optimize Triangulations via Bistellar Flips](https://mlsanity.com/paper/2605.30220/) (2026)
- [The Two-Hump Problem: Bridging the Difficulty Gap in Mathematical Reinforcement Learning](https://mlsanity.com/paper/2606.21611/) (2026)
- [SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance](https://mlsanity.com/paper/2609.30192/) (2026)

## Abstract (arXiv)

Using a long-standing conjecture from combinatorial group theory, we explore, from multiple perspectives, the challenges of finding rare instances carrying disproportionately high rewards. Based on lessons learned in the context defined by the Andrews-Curtis conjecture, we propose algorithmic enhancements and a topological hardness measure with implications for a broad class of search problems. As part of our study, we also address several open mathematical questions. Notably, we demonstrate the length reducibility of all but two presentations in the Akbulut-Kirby series (1981), and resolve various potential counterexamples in the Miller-Schupp series (1991), including three infinite subfamilies.