# RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems

arXiv: https://arxiv.org/abs/2510.02263 · HTML: https://mlsanity.com/paper/2510.02263/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 30 of 46)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)

## Built on by (most cited first, 21 of 21)

- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [TritonRL: Training LLMs to Think and Code Triton Without Cheating](https://mlsanity.com/paper/2510.17891/) (2025)
- [Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes](https://mlsanity.com/paper/2601.18795/) (2026)
- [Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL](https://mlsanity.com/paper/2602.03773/) (2026)
- [Expanding LLM Agent Boundaries with Strategy-Guided Exploration](https://mlsanity.com/paper/2603.02045/) (2026)
- [KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance](https://mlsanity.com/paper/2604.12627/) (2026)
- [Test-time Recursive Thinking: Self-Improvement without External Feedback](https://mlsanity.com/paper/2602.03094/) (2026)
- [Learning Next Action Predictors from Human-Computer Interaction](https://mlsanity.com/paper/2603.05923/) (2026)
- [Procedural Knowledge at Scale Improves Reasoning](https://mlsanity.com/paper/2604.01348/) (2026)
- [Do explanations generalize across large reasoning models?](https://mlsanity.com/paper/2601.11517/) (2026)
- [Test-Time Learning with an Evolving Library](https://mlsanity.com/paper/2605.14477/) (2026)
- [Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier](https://mlsanity.com/paper/2606.18284/) (2026)
- [From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory](https://mlsanity.com/paper/2606.08656/) (2026)
- [TCA-SIR: Learning Target-Conditioned Abstractions for Scientific Inspiration Retrieval](https://mlsanity.com/paper/2607.28498/) (2026)
- [Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Cost](https://mlsanity.com/paper/2608.11338/) (2026)
- [Inducing Reasoning Primitives from Agent Traces](https://mlsanity.com/paper/2606.02994/) (2026)
- [Notes to Self: Can LLMs Benefit from Experiential Abstractions?](https://mlsanity.com/paper/2607.20372/) (2026)
- [Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching](https://mlsanity.com/paper/2609.07303/) (2026)
- [Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning](https://mlsanity.com/paper/2609.26704/) (2026)
- [Mathematics for and by human cognition: A resource-rational search for bottlenecks in problem-solving](https://mlsanity.com/paper/2609.34410/) (2026)

## Abstract (arXiv)

Reasoning requires going beyond pattern matching or memorization of solutions to identify and implement "algorithmic procedures" that can be used to deduce answers to hard problems. Doing so requires realizing the most relevant primitives, intermediate results, or shared procedures, and building upon them. While RL post-training on long chains of thought ultimately aims to uncover this kind of algorithmic behavior, most reasoning traces learned by large models fail to consistently capture or reuse procedures, instead drifting into verbose and degenerate exploration. To address more effective reasoning, we introduce reasoning abstractions: concise natural language descriptions of procedural and factual knowledge that guide the model toward learning successful reasoning. We train models to be capable of proposing multiple abstractions given a problem, followed by RL that incentivizes building a solution while using the information provided by these abstractions. This results in a two-player RL training paradigm, abbreviated as RLAD, that jointly trains an abstraction generator and a solution generator. This setup effectively enables structured exploration, decouples learning signals of abstraction proposal and solution generation, and improves generalization to harder problems. We also show that allocating more test-time compute to generating abstractions is more beneficial for performance than generating more solutions at large test budgets, illustrating the role of abstractions in guiding meaningful exploration.