mlsanityData to 7 Oct 2026

How Does RL Post-training Induce Skill Composition? A Case Study on Countdown

arXiv 2512.01775 ↗First posted 1 Dec 2025cs.LGcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

While reinforcement learning (RL) successfully enhances reasoning in large language models, its role in fostering compositional generalization (the ability to synthesize novel skills from known components) is often conflated with mere length generalization. To this end, we study what RL post-training teaches about skill composition and how the structure of the composition affects the skill transfer. We focus on the Countdown task (given n numbers and a target, form an expression that evaluates to the target) and analyze model solutions as expression trees, where each subtree corresponds to a reusable subtask and thus can be viewed as a ``skill.'' Tracking tree shapes and their success rates over training, we find: (i) out-of-distribution (OOD) generalization to larger n and to unseen tree shapes, indicating compositional reuse of subtasks; (ii) a structure-dependent hierarchy of learnability -- models master shallow balanced trees (workload is balanced between subtasks) before deep unbalanced ones, with persistent fragility on right-heavy structures (even when the composition depth is the same as some left-heavy structures). Our diagnostic reveals what is learned, in what order, and where generalization fails, clarifying how RL-only post-training induces OOD generalization beyond what standard metrics such as pass@k reveal.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 35 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

26 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2022Exploring Length Generalization in Large Language Modelscited by 204
  7. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  8. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  9. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  10. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  11. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  12. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
9 more
  1. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  2. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  3. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 48
  4. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
  5. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  6. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  7. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  8. 2024Can Models Learn Skill Composition from Examples?cited by 17
  9. 2025LLM-First Search: Self-Guided Exploration of the Solution Spacecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 2
  2. 2026Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Trainingcited by 2
  3. 2026New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVRcited by 1
  4. 2026Characterizing Model-Native Skillscited by 0
  5. 2026Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficientlycited by 0
  6. 2026Toward Latent Language Model Skills Steering and Optimization: An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Compositional Reasoning in Language Models under Reinforcement Learning Post-Training
  2. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  3. 2026Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
  4. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions
  5. 2026Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
  6. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  7. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
  8. 2025Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?