# Learning Dynamics of VLM Finetuning

arXiv: https://arxiv.org/abs/2510.11978 · HTML: https://mlsanity.com/paper/2510.11978/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization](https://mlsanity.com/paper/2411.02712/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key](https://mlsanity.com/paper/2501.09695/) (2025)
- [CF-VLM:CounterFactual Vision-Language Fine-tuning](https://mlsanity.com/paper/2506.17267/) (2025)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)

## Built on by (most cited first, 17 of 17)

- [RaCoT: Plug-and-Play Contrastive Example Generation Mechanism for Enhanced LLM Reasoning Reliability](https://mlsanity.com/paper/2510.22710/) (2025)
- [STORM: Search-Guided Generative World Models for Robotic Manipulation](https://mlsanity.com/paper/2512.18477/) (2025)
- [Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models](https://mlsanity.com/paper/2512.18496/) (2025)
- [AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents](https://mlsanity.com/paper/2602.15325/) (2026)
- [Agent-GSPO: Communication-Efficient Multi-Agent Systems via Group Sequence Policy Optimization](https://mlsanity.com/paper/2510.22477/) (2025)
- [Reflective Confidence: Correcting Reasoning Flaws via Online Self-Correction](https://mlsanity.com/paper/2512.18605/) (2025)
- [DynaPose4D: High-Quality 4D Dynamic Content Generation via Pose Alignment Loss](https://mlsanity.com/paper/2510.22473/) (2025)
- [Guardian: Decoupling Exploration from Safety in Reinforcement Learning](https://mlsanity.com/paper/2510.22859/) (2025)
- [GTMA: Dynamic Representation Optimization for OOD Vision-Language Models](https://mlsanity.com/paper/2512.18504/) (2025)
- [PTTA: A Pure Text-to-Animation Framework for High-Quality Creation](https://mlsanity.com/paper/2512.18614/) (2025)
- [SirenPose: Dynamic Scene Reconstruction via Geometric Supervision](https://mlsanity.com/paper/2512.20531/) (2025)
- [RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks](https://mlsanity.com/paper/2512.20920/) (2025)
- [A Three-Level Alignment Framework for Large-Scale 3D Retrieval and Controlled 4D Generation](https://mlsanity.com/paper/2512.22294/) (2025)
- [Learning Dynamic Scene Reconstruction with Sinusoidal Geometric Priors](https://mlsanity.com/paper/2512.22295/) (2025)
- [Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains](https://mlsanity.com/paper/2512.22545/) (2025)
- [Forgetting as a Feature: Cognitive Alignment of Large Language Models](https://mlsanity.com/paper/2601.09726/) (2025)
- [Multi-Agent Cooperative Learning for Robust Vision-Language Alignment under OOD Concepts](https://mlsanity.com/paper/2601.09746/) (2026)

## Abstract (arXiv)

Preference-based finetuning of vision--language models (VLMs) is brittle: trivially wrong negatives inject uninformative gradients that destabilize training. We recast alignment as learning-dynamics--aware optimization and introduce Cooling-Weighted DPO (CW-DPO), a two-stage recipe that explicitly models and exploits the training trajectory. Stage 1 performs supervised finetuning with gentle negatives: low-weight smoothed supervision that regularizes the base policy and curbs overconfidence without explicit penalties. Stage 2 applies a DPO objective in which the negative term is scaled by a cooling weight computed from the model's average token log-probability on each negative, suppressing uninformative gradients from easy or off-distribution samples while preserving signal from hard negatives. In practice, we emphasize on-policy negatives and allow mixed negatives by blending a controllable fraction of dataset negatives to maintain contrast freshness. Throughout, we instrument training with Delta!log p probes on positives and negatives as first-class signals for early stopping, curriculum design, and failure diagnosis. Across diverse VLM tasks, CW-DPO yields more stable optimization, better calibration, and higher pairwise win-rates than SFT-only and vanilla DPO, while converging in fewer steps. Ablations isolate the cooling-weight mechanism as the primary driver of these gains and show complementary benefits from mixing on-policy and dataset negatives. Taken together, our results show that smoothing learning dynamics before cooling preferences is a simple, general principle for robust VLM alignment.