# Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents

arXiv: https://arxiv.org/abs/2610.05195 · HTML: https://mlsanity.com/paper/2610.05195/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [MMSearch-R1: Incentivizing LMMs to Search](https://mlsanity.com/paper/2506.20670/) (2025)
- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines](https://mlsanity.com/paper/2409.12959/) (2024)
- [SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models](https://mlsanity.com/paper/2502.13059/) (2025)
- [REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents](https://mlsanity.com/paper/2602.14234/) (2026)
- [Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2601.22060/) (2026)
- [OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents](https://mlsanity.com/paper/2605.05185/) (2026)
- [VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2603.16289/) (2026)
- [Towards Long-horizon Agentic Multimodal Search](https://mlsanity.com/paper/2604.12890/) (2026)
- [HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents](https://mlsanity.com/paper/2605.07177/) (2026)
- [VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning](https://mlsanity.com/paper/2603.02795/) (2026)
- [BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2602.12876/) (2026)
- [SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation](https://mlsanity.com/paper/2607.05943/) (2026)
- [DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents](https://mlsanity.com/paper/2608.01827/) (2026)
- [WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents](https://mlsanity.com/paper/2608.28062/) (2026)

## Abstract (arXiv)

The next generation of multimodal research agents must reason over long-lived research histories rather than short model completions. During a single task, an agent may repeatedly search the web, inspect visual evidence, revisit earlier hypotheses, and accumulate tens of thousands of tokens of multimodal context. Despite this trend, online RL for multimodal research agents remains largely confined to shorter contexts and interaction horizons. We push online RL training to 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context, and the first trained with a 75 tool-turn horizon. Scaling to this regime exposes several practical limitations of conventional RL training. Early in training, weak policies make poor use of large interaction budgets, causing expensive rollouts with little reward improvement. Later, policy entropy can collapse before performance has saturated, prematurely ending useful learning. We introduce Long-MDR, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques improve both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly. At a 50-turn evaluation budget, our RL-trained Long-MDR-9B ranks first on five of six benchmarks among the compared 7B-9B agents.