mlsanityData to 7 Oct 2026

GIFT: Reconciling Post-Training Objectives via Variational Finite-Temperature Gibbs Initialization

arXiv 2601.09233 ↗First posted 14 Jan 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The prevailing post-training paradigm for Large Reasoning Models (LRMs)---Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL)-suffers from an intrinsic optimization mismatch: the rigid likelihood maximization in SFT induces distributional collapse, thereby exhausting the exploration space necessary for subsequent RL. Motivated by the Gibbs optimum of KL-regularized RL, we derive a token-level variational surrogate that makes the SFT target structurally compatible with the subsequent RL stage, and propose Gibbs Initialization with Finite Temperature (GIFT). Standard SFT emerges as a degenerate zero-temperature limit of this surrogate, while a finite temperature preserves structural diversity. Our experiments demonstrate that GIFT outperforms standard SFT and surpasses competitive baselines when utilized for RL initialization. Our code is available at https://github.com/zzy1127/GIFT.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 36 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

31 of 36 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead2 Oct
  2. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning23 May
  6. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  7. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
  8. 2025Learning to Reason under Off-Policy Guidance21 Apr
  9. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2023The False Promise of Imitating Proprietary LLMscited by 254
8 more
  1. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  3. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  4. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  5. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18
  6. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  7. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
  8. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5
  2. 2026ANDES: Agent Native Data Evolving Synthesis Tool for Autonomous Instruction Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  2. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
  3. 2026Unifying ICL, SFT, KL-Regularized RL Through a Bayesian Lens
  4. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  5. 2025Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
  6. 2025A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
  7. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
  8. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning