Stage-1 Controls the Entropy Regime, Not the Outcome
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs). We ask what Stage-1 actually controls in a small-data study using Qwen2.5-VL-7B with a same-modality 72B VLM teacher for OPD. First, the three warm-starts reach a narrow 53--54% band on Geometry3K internal validation, consistent with the narrow range reported by recent specialized methods; this setup provides little evidence that Stage-1 changes the in-domain endpoint. Second, a matched-recipe, early-stopped SFT improves out-of-domain MathVista by +2.1 points, reversing the -9.5-point drop of an over-trained variant. The clearest difference is the entropy regime: OPD enters RL with substantially higher policy entropy than either SFT initialization, and the separation remains visible through the available trajectories. At the in-domain initialization, OPD also has higher answer diversity and pass@16 (+2.0 to +5.2 points over SFT), although problem-level bootstrap intervals show that the smaller contrast is uncertain. The advantage is absent after RL (endpoint pass@16 values within 1.1 points) and on MathVista (six models within 1.2 points). Our contribution is therefore a bounded empirical characterization: Stage-1 is strongly associated with the entropy regime in this setup, but the downstream payoff is small, localized, and not evidence that OPD is a better RL warm-start.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 8 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
8 of 8 references.
Closest ancestors · 2
Same direction, newest first
- 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation27 Oct
- 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning10 Mar
Other references · 6
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2025Geometric-Mean Policy Optimizationcited by 107
- 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
- 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
- 2026Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillationcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
- 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
- 2026Entropy-Preserving Reinforcement Learning
- 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
- 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
- 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control