# Stage-1 Controls the Entropy Regime, Not the Outcome

arXiv: https://arxiv.org/abs/2606.09059 · HTML: https://mlsanity.com/paper/2606.09059/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 8 of its 8 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)
- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2503.07065/) (2025)
- [OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe](https://mlsanity.com/paper/2511.16334/) (2025)
- [VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation](https://mlsanity.com/paper/2510.23497/) (2025)
- [Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation](https://mlsanity.com/paper/2605.16826/) (2026)
- [Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation](https://mlsanity.com/paper/2605.22731/) (2026)

## Built on by (most cited first, 1 of 1)

- [Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training](https://mlsanity.com/paper/2609.31900/) (2026)

## Abstract (arXiv)

Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs). We ask what Stage-1 actually controls in a small-data study using Qwen2.5-VL-7B with a same-modality 72B VLM teacher for OPD. First, the three warm-starts reach a narrow 53--54% band on Geometry3K internal validation, consistent with the narrow range reported by recent specialized methods; this setup provides little evidence that Stage-1 changes the in-domain endpoint. Second, a matched-recipe, early-stopped SFT improves out-of-domain MathVista by +2.1 points, reversing the -9.5-point drop of an over-trained variant. The clearest difference is the entropy regime: OPD enters RL with substantially higher policy entropy than either SFT initialization, and the separation remains visible through the available trajectories. At the in-domain initialization, OPD also has higher answer diversity and pass@16 (+2.0 to +5.2 points over SFT), although problem-level bootstrap intervals show that the smaller contrast is uncertain. The advantage is absent after RL (endpoint pass@16 values within 1.1 points) and on MathVista (six models within 1.2 points). Our contribution is therefore a bounded empirical characterization: Stage-1 is strongly associated with the entropy regime in this setup, but the downstream payoff is small, localized, and not evidence that OPD is a better RL warm-start.