mlsanityData to 14 Oct 2026

NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

arXiv 2504.13055 ↗First posted 17 Apr 2025cs.CVcited by 86

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to better scale test-time compute remains largely underexplored. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. We introduce NoisyRollout, a simple yet effective data augmentation method that addresses these issues by mixing training trajectories from both clean and moderately distorted images. This approach injects perceptual diversity, encouraging better policy exploration and leading to more robust reasoning. A noise annealing schedule gradually reduces distortion strength, aiding exploration early in training while ensuring later stability. Crucially, our method is easy-to-adopt--requiring no additional training cost and no modifications to the RL objective. Extensive experiments on 2 distinct training datasets demonstrate that NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models across 5 out-of-domain reasoning and perception benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes (7B and 32B), data scales (from 1K to 6K) and image augmentation types (Gaussion noise and rotation), highlighting its generalizability and scalability.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 70 of its 93 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

70 of 93 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models10 Apr
  2. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement10 Apr
  3. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme3 Apr
  4. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  5. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoning13 Mar
  6. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
  7. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning10 Mar
  8. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  9. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement9 Mar
  10. 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
  11. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency13 Feb
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
7 more
  1. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models21 Nov
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  3. 2024Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning16 Aug
  4. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  5. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  6. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2024GPT-4o System Cardcited by 4,588
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  11. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  14. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  15. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  16. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  17. 2025Process Reinforcement through Implicit Rewardscited by 504
  18. 2020Reinforcement Learning with Augmented Datacited by 468
  19. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  20. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  21. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  22. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  23. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  24. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  25. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  26. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  27. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  28. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  29. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  30. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  31. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  32. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 98
  33. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 86
  34. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  35. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  36. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Taskscited by 41
  37. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  38. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 42.

86 papers.

Sort:

By area:

  1. 2025Agentic Reinforced Policy Optimizationcited by 172
  2. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  3. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  4. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  5. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  6. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  7. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
  8. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  9. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  10. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  11. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  12. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
38 more
  1. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 24
  2. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  3. 2025WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learningcited by 21
  4. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  5. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  6. 2025M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoningcited by 16
  7. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  8. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  9. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14
  10. 2025Sherlock: Self-Correcting Reasoning in Vision-Language Modelscited by 13
  11. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  12. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  13. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  14. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  15. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  16. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 11
  17. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  18. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10
  19. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  20. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  21. 2025Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoningcited by 9
  22. 2025MiCo: Multi-image Contrast for Reinforcement Visual Reasoningcited by 9
  23. 2025Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoningcited by 9
  24. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
  25. 2026Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoningcited by 9
  26. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  27. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  28. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
  29. 2025Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoningcited by 7
  30. 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Modelscited by 7
  31. 2025Video Spatial Reasoning with Object-Centric 3D Rolloutcited by 6
  32. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  33. 2026Self-Supervised Visual On-Policy Distillationcited by 6
  34. 2025Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoningcited by 5
  35. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  36. 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshapingcited by 5
  37. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  38. 2025VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Groundingcited by 4

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
  2. 2026Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models
  3. 2026What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis
  4. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
  5. 2025NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
  6. 2026Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models
  7. 2026Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning
  8. 2026Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning