# VisPlay: Self-Evolving Vision-Language Models from Images

arXiv: https://arxiv.org/abs/2511.15661 · HTML: https://mlsanity.com/paper/2511.15661/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 30 of 39)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization](https://mlsanity.com/paper/2503.12937/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL](https://mlsanity.com/paper/2503.07536/) (2025)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.14677/) (2025)
- [AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence](https://mlsanity.com/paper/1905.10985/) (2019)
- [SPICE: Self-Play In Corpus Environments Improves Reasoning](https://mlsanity.com/paper/2510.24684/) (2025)
- [Efficient Test-Time Scaling via Self-Calibration](https://mlsanity.com/paper/2503.00031/) (2025)
- [Parallel-R1: Towards Parallel Thinking via Reinforcement Learning](https://mlsanity.com/paper/2509.07980/) (2025)
- [One Token to Fool LLM-as-a-Judge](https://mlsanity.com/paper/2507.08794/) (2025)
- [Multi-Agent Evolve: LLM Self-Improve through Co-evolution](https://mlsanity.com/paper/2510.23595/) (2025)
- [SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward](https://mlsanity.com/paper/2505.17018/) (2025)
- [Language Self-Play For Data-Free Training](https://mlsanity.com/paper/2509.07414/) (2025)
- [Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play](https://mlsanity.com/paper/2509.25541/) (2025)
- [Search Self-play: Pushing the Frontier of Agent Capability without Supervision](https://mlsanity.com/paper/2510.18821/) (2025)
- [Weak-to-Strong Reasoning](https://mlsanity.com/paper/2407.13647/) (2024)
- [Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning](https://mlsanity.com/paper/2505.13886/) (2025)
- [CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models](https://mlsanity.com/paper/2509.09675/) (2025)

## Built on by (most cited first, 30 of 47)

- [Guided Self-Evolving LLMs with Minimal Human Supervision](https://mlsanity.com/paper/2512.02472/) (2025)
- [Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning](https://mlsanity.com/paper/2511.19900/) (2025)
- [MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Data](https://mlsanity.com/paper/2603.09206/) (2026)
- [TTCS: Test-Time Curriculum Synthesis for Self-Evolving](https://mlsanity.com/paper/2601.22628/) (2026)
- [Recursive Synthesis for Long-Horizon Terminal Tasks](https://mlsanity.com/paper/2608.05466/) (2026)
- [G-Zero: Self-Play for Open-Ended Generation from Zero Data](https://mlsanity.com/paper/2605.09959/) (2026)
- [Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2604.20987/) (2026)
- [EnvHarness: Awakening Static Worlds for Agent Learning](https://mlsanity.com/paper/2608.19880/) (2026)
- [Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning](https://mlsanity.com/paper/2601.18984/) (2026)
- [Visually-Guided Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2604.09349/) (2026)
- [R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training](https://mlsanity.com/paper/2602.13103/) (2026)
- [When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning](https://mlsanity.com/paper/2602.10560/) (2026)
- [Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration](https://mlsanity.com/paper/2602.11241/) (2026)
- [iReasoner: Trajectory-Aware Intrinsic Reasoning Supervision for Self-Evolving Large Multimodal Models](https://mlsanity.com/paper/2601.05877/) (2026)
- [Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution](https://mlsanity.com/paper/2601.22528/) (2026)
- [Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models](https://mlsanity.com/paper/2606.27373/) (2026)
- [Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Tasks](https://mlsanity.com/paper/2601.18226/) (2026)
- [SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments](https://mlsanity.com/paper/2604.14144/) (2026)
- [Adaptive Time Series Reasoning via Segment Selection](https://mlsanity.com/paper/2602.18645/) (2026)
- [EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations](https://mlsanity.com/paper/2604.18320/) (2026)
- [EvoGround: Self-Evolving Video Agents for Video Temporal Grounding](https://mlsanity.com/paper/2605.13803/) (2026)
- [CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphs](https://mlsanity.com/paper/2602.07080/) (2026)
- [Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling](https://mlsanity.com/paper/2604.03647/) (2026)
- [EvoVid: Temporal-Centric Self-Evolution for Video Large Language Models](https://mlsanity.com/paper/2605.21931/) (2026)
- [Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning](https://mlsanity.com/paper/2607.20166/) (2026)
- [TTSR: Test-Time Self-Evolving via Reflection](https://mlsanity.com/paper/2603.03297/) (2026)
- [Structured Role-Aware Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2605.07274/) (2026)
- [DUEL: Adversarial Self-Play for Multimodal Reasoning](https://mlsanity.com/paper/2605.24794/) (2026)
- [Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots](https://mlsanity.com/paper/2608.09931/) (2026)
- [AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedback](https://mlsanity.com/paper/2602.03084/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) provides a principled framework for improving Vision-Language Models (VLMs) on complex reasoning tasks. However, existing RL approaches often rely on human-annotated labels or task-specific heuristics to define verifiable rewards, both of which are costly and difficult to scale. We introduce VisPlay, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning abilities using large amounts of unlabeled image data. Starting from a single base VLM, VisPlay assigns the model into two interacting roles: an Image-Conditioned Questioner that formulates challenging yet answerable visual questions, and a Multimodal Reasoner that generates silver responses. These roles are jointly trained with Group Relative Policy Optimization (GRPO), which incorporates diversity and difficulty rewards to balance the complexity of generated questions with the quality of the silver answers. VisPlay scales efficiently across two model families. When trained on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements in visual reasoning, compositional generalization, and hallucination reduction across eight benchmarks, including MM-Vet and MMMU, demonstrating a scalable path toward self-evolving multimodal intelligence. The project page is available at https://bruno686.github.io/VisPlay/