# Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

arXiv: https://arxiv.org/abs/2609.16059 · HTML: https://mlsanity.com/paper/2609.16059/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 13 Sep 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-play RL for reasoning](https://mlsanity.com/synth/self-play-rl-for-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs](https://mlsanity.com/paper/2406.11833/) (2024)
- [Preserving Diversity in Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2408.16673/) (2024)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2407.01509/) (2024)
- [Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay](https://mlsanity.com/paper/2506.05316/) (2025)
- [VerIF: Verification Engineering for Reinforcement Learning in Instruction Following](https://mlsanity.com/paper/2506.09942/) (2025)
- [MM-IFEngine: Towards Multimodal Instruction Following](https://mlsanity.com/paper/2504.07957/) (2025)
- [MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity](https://mlsanity.com/paper/2407.15838/) (2024)
- [OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference](https://mlsanity.com/paper/2502.18411/) (2025)
- [TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild](https://mlsanity.com/paper/2309.08637/) (2023)
- [CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World](https://mlsanity.com/paper/2505.11962/) (2025)
- [AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints](https://mlsanity.com/paper/2603.13348/) (2026)
- [Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following](https://mlsanity.com/paper/2601.04954/) (2026)

## Abstract (arXiv)

Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (Multimodal Instruction Following Synthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13% on four MMIF benchmarks and a 3× faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.