mlsanityData to 7 Oct 2026

Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspective

arXiv 2601.14599 ↗First posted 21 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

A large number of heuristics have been proposed to optimize the reinforcement fine-tuning of LLMs. However, inconsistent claims are made from time to time, making this area elusive. Reflecting on this situation, two fundamental questions still lack a clear understanding: 1) what is the role of each optimizing choice? 2) which ones are the bottlenecks? This paper aims to shed light on them, and it faces the challenge of several entangled confounding factors in the fine-tuning process. To tackle this challenge, we propose a bottom-up experiment pipeline. The bottom layer is composed of a minimalist configuration: one training data, one rollout per round and the reward directly serve as the learning signal without advantage function design. This minimalist configuration connects to multi-armed bandit learning with extremely large discrete action space, which offers theories to corroborate the experiment findings. The up procedure of the experiment pipeline expanding the minimalist configuration layer by layer, examining the role of each design choice. Experimental results on three LLMs and two reasoning datasets not only reveal new understanding of the design choice but also yield essential insights to shape the area.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 48 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

24 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward18 Dec
  2. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO29 Sep
  3. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning26 Sep
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  5. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration19 Aug
  6. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets15 Aug
  7. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy6 Aug
  8. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  9. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  10. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  11. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 20242 OLMo 2 Furiouscited by 302
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  8. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  9. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  10. 2024Scaling Law for Language Models Training Considering Batch Sizecited by 19
  11. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  12. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
  2. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
  3. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
  4. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  5. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  6. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  7. 2026Rethinking Reinforcement Fine-Tuning in LVLM: Convergence, Reward Decomposition, and Generalization
  8. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities