UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of large language models (LLMs) on mathematics and programming tasks, but standard approaches that optimize single-attempt accuracy can inadvertently suppress response diversity across repeated attempts, narrowing exploration and overlooking underrepresented strategies. We introduce UpSkill, a training time method that adapts Mutual Information Skill Learning (MISL) to LLMs for optimizing pass@k correctness. We propose a novel reward that we implement within Group Relative Policy Optimization (GRPO): a token-level mutual information (MI) reward that encourages trajectory specificity to z. Experiments on GSM8K with three open-weight models, Llama 3.1-8B, Qwen 2.5-7B, and R1-Distilled-Qwen2.5-Math-1.5B, show that UpSkill improves multi-attempt metrics on the stronger base models, yielding mean gains of ~3% in pass@k for both Qwen and Llama without degrading pass@1. Additionally, we find both empirical and theoretical evidence that improvements in pass@k are closely tied to the mutual information objective.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 58 references that are in the map. First posted on arXiv 25 Feb 2026.
Builds on
41 of 58 references.
Closest ancestors · 2
Same direction, newest first
- 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 39
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2018Hierarchical Neural Story Generationcited by 1,182
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
- 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
27 more
- 2016Variational Intrinsic Controlcited by 262
- 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
- 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
- 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
- 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
- 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
- 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
- 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
- 2017InfoVAE: Information Maximizing Variational Autoencoderscited by 124
- 2018Variational Option Discovery Algorithmscited by 121
- 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
- 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
- 2022On Decoding Strategies for Neural Text Generatorscited by 89
- 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
- 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
- 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
- 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
- 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
- 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 28
- 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
- 2021Determinantal Beam Searchcited by 18
- 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
- 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
- 2024Growing a Tail: Increasing Output Diversity in Large Language Modelscited by 9
- 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
- 2020Learning Discrete Structured Representations by Adversarially Maximizing Mutual Informationcited by 5
- 2024Asking Again and Again: Exploring LLM Robustness to Repeated Questionscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
- 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
- 2025Bridging Offline and Online Reinforcement Learning for LLMs
- 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
- 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
- 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
- 2025Outcome-based Exploration for LLM Reasoning
- 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward