# From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones

arXiv: https://arxiv.org/abs/2509.25123 · HTML: https://mlsanity.com/paper/2509.25123/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining](https://mlsanity.com/paper/2504.07912/) (2025)
- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening](https://mlsanity.com/paper/2506.02355/) (2025)
- [Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective](https://mlsanity.com/paper/2506.14965/) (2025)
- [OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization](https://mlsanity.com/paper/2506.18880/) (2025)
- [Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.08904/) (2018)
- [Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions](https://mlsanity.com/paper/2508.21188/) (2025)
- [Learning Composable Chains-of-Thought](https://mlsanity.com/paper/2505.22635/) (2025)

## Built on by (most cited first, 30 of 42)

- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare](https://mlsanity.com/paper/2602.06717/) (2026)
- [h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning](https://mlsanity.com/paper/2510.07312/) (2025)
- [Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies](https://mlsanity.com/paper/2512.01970/) (2025)
- [Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning](https://mlsanity.com/paper/2601.15160/) (2026)
- [Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs](https://mlsanity.com/paper/2602.07729/) (2026)
- [How Does RL Post-training Induce Skill Composition? A Case Study on Countdown](https://mlsanity.com/paper/2512.01775/) (2025)
- [Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures](https://mlsanity.com/paper/2601.19928/) (2026)
- [Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models](https://mlsanity.com/paper/2602.12036/) (2026)
- [Learning from Synthetic Data Improves Multi-hop Reasoning](https://mlsanity.com/paper/2603.02091/) (2026)
- [On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length](https://mlsanity.com/paper/2605.02572/) (2026)
- [Debugging code world models](https://mlsanity.com/paper/2602.07672/) (2026)
- [From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning](https://mlsanity.com/paper/2606.18089/) (2026)
- [Understanding Reasoning from Pretraining to Post-Training](https://mlsanity.com/paper/2607.16097/) (2026)
- [GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR](https://mlsanity.com/paper/2601.09361/) (2026)
- [Text Before Vision: Staged Knowledge Injection Matters for Agentic RLVR in Ultra-High-Resolution Remote Sensing Understanding](https://mlsanity.com/paper/2602.14225/) (2026)
- [Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems](https://mlsanity.com/paper/2604.04767/) (2026)
- [The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation](https://mlsanity.com/paper/2607.24720/) (2026)
- [Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Models](https://mlsanity.com/paper/2601.18129/) (2026)
- [New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR](https://mlsanity.com/paper/2602.08281/) (2026)
- [Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training](https://mlsanity.com/paper/2603.02208/) (2026)
- [Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.06326/) (2026)
- [Select and Improve: Understanding the Mechanics of Post-Training for Reasoning](https://mlsanity.com/paper/2606.13125/) (2026)
- [RL Post-Training Builds Compositional Reasoning Strategies](https://mlsanity.com/paper/2607.07646/) (2026)
- [Learning Generalizable Behaviors for Terminal Agents](https://mlsanity.com/paper/2608.22631/) (2026)
- [When Is Compositional Reasoning Learnable from Verifiable Rewards?](https://mlsanity.com/paper/2602.07992/) (2026)
- [Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States](https://mlsanity.com/paper/2603.19987/) (2026)
- [Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models](https://mlsanity.com/paper/2605.08472/) (2026)

## Abstract (arXiv)

Does RL teach LLMs genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role of RL in LLM post-training. On one side, strong empirical results can be achieved with RL even without preceding supervised finetuning; on the other, critics argue that RL contributes little beyond reweighting existing reasoning strategies. This work provides concrete evidence that LLMs can acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills. To mitigate data contamination and other confounding factors, and to allow precise control over task complexity, we develop a synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the output of a string transformation function f(x) given x. When an LLM has already learned f and g prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them h(x)=g(f(x)). Further, this compositional ability generalizes to more difficult problems such as compositions of >2 functions unseen during RL training. Surprisingly, our experiments show that compositional skill acquired on a source task transfers to a different target task. This transfer happens even without compositional training on the target, requiring only prior knowledge of the target's atomic skills. Our qualitative analysis shows that RL fundamentally changes the reasoning behaviors of the models. In contrast, next-token training with the same data yields none of these findings. Our systematic experiments provide fresh insights into LLM learning, suggesting the value of first building base models with basic skills, then using RL to incentivize advanced, generalizable skills for complex problems.