mlsanityData to 7 Oct 2026

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

arXiv 2604.25907 ↗First posted 28 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

SFT-then-RLVR is widely used for post-training reasoning models, but why this specific ordering, and why RLVR-only stalls at cold start, have lacked a unifying theoretical account. We provide that account under a unified loss family JQ using the Tsallis q-logarithm. JQ is a single-parameter family that interpolates between RLVR (at q=0, the textitexploitation pole) and the log-marginal-likelihood over latent trajectories (at q=1, the textitdensity-estimation pole), under which the standard pipeline corresponds to a stepwise q=1 → 0 schedule. All members share the same per-example gradient direction, differing only by a per-instance amplification Pθ-q that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires Omega(frac1p0) time to escape cold start but is robust to label noise, while the density-estimation pole escapes in Thetabig(log(frac1p0)big) but memorizes label noise. This separation explains how SFT (q=1) first moves the model out of the cold-start regime, followed by the more robust RLVR (q=0), under the SFT-then-RLVR paradigm. We further derive two Monte Carlo estimators that directly optimize fixed-q on the JQ continuum, without annotated rationales: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias Obig(fracqM Pθqbig) but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high q substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low q dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes and PAFT at q=0.75 remains stable, reaching 47.9 textttm@16 on HotPotQA (+13.9 over GRPO).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 42 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

29 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoning14 Dec
  2. 2025Reinforcing General Reasoning without Verifiers27 May
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  8. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  9. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
13 more
  1. 2016R'enyi Divergence Variational Inferencecited by 104
  2. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  3. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  4. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  5. 2020Limitations of Autoregressive Models and Their Alternativescited by 54
  6. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 46
  7. 2017From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihoodcited by 41
  8. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  9. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  10. 2026Maximum Likelihood Reinforcement Learningcited by 29
  11. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  12. 2017Cold-Start Reinforcement Learning with Softmax Policy Gradientcited by 15
  13. 2026Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectivescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
  2. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  3. 2026You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
  4. 2025Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards
  5. 2025The Path Not Taken: RLVR Provably Learns Off the Principals
  6. 2026Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
  7. 2026Detecting RLVR Training Data via Structural Convergence of Reasoning
  8. 2026Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging