# DUEL: Adversarial Self-Play for Multimodal Reasoning

arXiv: https://arxiv.org/abs/2605.24794 · HTML: https://mlsanity.com/paper/2605.24794/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 24 May 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback](https://mlsanity.com/paper/2402.03681/) (2024)
- [OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles](https://mlsanity.com/paper/2503.17352/) (2025)
- [VisPlay: Self-Evolving Vision-Language Models from Images](https://mlsanity.com/paper/2511.15661/) (2025)
- [Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play](https://mlsanity.com/paper/2509.25541/) (2025)
- [EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards](https://mlsanity.com/paper/2511.16672/) (2025)

## Built on by (most cited first, 2 of 2)

- [Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models](https://mlsanity.com/paper/2609.36224/) (2026)
- [ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensing](https://mlsanity.com/paper/2610.18388/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has emerged as an effective paradigm for improving the reasoning capability of vision-language models (VLMs). However, RL-based optimization typically depends on costly high-quality annotations that are difficult to scale. Existing unsupervised alternatives may drift toward biased solutions due to weak visual grounding and the lack of reliable verification signals. We propose a self-evolving post-training framework, DUEL, where supervision emerges from adversarial interactions between two policies initialized from the same pretrained VLM. A Challenger generates an image-grounded true claim together with a minimally perturbed hard-negative counterpart, while a Solver verifies both claims against the image, encouraging fine-grained visual discrimination under near-neighbor semantics. To stabilize optimization, we introduce a length-normalized log-likelihood reward that preserves informative optimization signals beyond binary outcome supervision and improves learning stability under sparse feedback. Experiments show that DUEL consistently improves visual reasoning and robust discrimination without additional human annotations, external reward models, or image editing tools.