# Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

arXiv: https://arxiv.org/abs/2507.20766 · HTML: https://mlsanity.com/paper/2507.20766/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 58 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning](https://mlsanity.com/paper/2505.14362/) (2025)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.15966/) (2025)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning](https://mlsanity.com/paper/2505.08617/) (2025)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [MiMo-VL Technical Report](https://mlsanity.com/paper/2506.03569/) (2025)
- [ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation](https://mlsanity.com/paper/2501.06598/) (2025)
- [VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use](https://mlsanity.com/paper/2505.19255/) (2025)
- [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding](https://mlsanity.com/paper/2501.05452/) (2025)
- [VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.22019/) (2025)

## Built on by (most cited first, 30 of 32)

- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [Reinforced Visual Perception with Tools](https://mlsanity.com/paper/2509.01656/) (2025)
- [Think3D: Thinking with Space for Spatial Reasoning](https://mlsanity.com/paper/2601.13029/) (2026)
- [VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning](https://mlsanity.com/paper/2511.00391/) (2025)
- [Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation](https://mlsanity.com/paper/2508.13587/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [SAIL-VL2 Technical Report](https://mlsanity.com/paper/2509.14033/) (2025)
- [HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better](https://mlsanity.com/paper/2607.04884/) (2026)
- [Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning](https://mlsanity.com/paper/2508.02258/) (2025)
- [Imagination Helps Visual Reasoning, But Not Yet in Latent Space](https://mlsanity.com/paper/2602.22766/) (2026)
- [Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding](https://mlsanity.com/paper/2603.13366/) (2026)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning](https://mlsanity.com/paper/2604.06079/) (2026)
- [PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs](https://mlsanity.com/paper/2512.14944/) (2025)
- [EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations](https://mlsanity.com/paper/2604.18320/) (2026)
- [Visual Reasoning through Tool-supervised Reinforcement Learning](https://mlsanity.com/paper/2604.19945/) (2026)
- [DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning](https://mlsanity.com/paper/2510.15050/) (2025)
- [Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model](https://mlsanity.com/paper/2605.12163/) (2026)
- [SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning](https://mlsanity.com/paper/2511.02280/) (2025)
- [CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution](https://mlsanity.com/paper/2604.22192/) (2026)
- [ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats](https://mlsanity.com/paper/2606.01348/) (2026)
- [OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents](https://mlsanity.com/paper/2607.09068/) (2026)
- [From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models](https://mlsanity.com/paper/2602.22859/) (2026)
- [Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning](https://mlsanity.com/paper/2603.14811/) (2026)
- [The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping](https://mlsanity.com/paper/2604.11297/) (2026)
- [Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning](https://mlsanity.com/paper/2606.07436/) (2026)
- [Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts](https://mlsanity.com/paper/2606.10334/) (2026)
- [Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding](https://mlsanity.com/paper/2606.13156/) (2026)
- [Thinking with Drafting: Optical Decompression via Logical Reconstruction](https://mlsanity.com/paper/2602.11731/) (2026)
- [From Pixels to PCells: A Neurosymbolic Approach to Photonic Component Creation](https://mlsanity.com/paper/2608.00084/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) exhibit impressive performance across various visual tasks. Subsequent investigations into enhancing their visual reasoning abilities have significantly expanded their performance envelope. However, a critical bottleneck in the advancement of MLLMs toward deep visual reasoning is their heavy reliance on curated image-text supervision. To solve this problem, we introduce a novel framework, ``Reasoning-Rendering-Visual-Feedback'' (RRVF), that enables MLLMs to learn complex visual reasoning from only raw images. This framework builds on the ``Asymmetry of Verification'' principle, i.e., verifying the rendered output against the source image is substantially easier than performing deep visual reasoning to generate a faithful, structured representation such as code. We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning (RL), thereby reducing reliance on image-text supervision. RRVF implements a closed-loop iterative process encompassing reasoning, rendering, and visual feedback components, enabling the model to perform complex reasoning, including self-correction through multi-turn interactions. This process is optimized end-to-end using the GRPO algorithm. Extensive evaluations are conducted on image-to-code generation across two diverse domains: data charts and web interfaces. The RRVF-trained model not only outperforms existing similarly sized open-source MLLMs and supervised fine-tuning baselines but also exhibits superior generalization. Notably, the model outperforms the more advanced MLLM used to generate visual feedback during training. Code is available at https://github.com/L-O-I/RRVF.