# ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL

arXiv: https://arxiv.org/abs/2602.22623 · HTML: https://mlsanity.com/paper/2602.22623/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 29 of 29)

- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning](https://mlsanity.com/paper/2310.01061/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://mlsanity.com/paper/2509.09674/) (2025)
- [SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement](https://mlsanity.com/paper/2504.07934/) (2025)
- [Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search](https://mlsanity.com/paper/2509.07969/) (2025)
- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [PyVision: Agentic Vision with Dynamic Tooling](https://mlsanity.com/paper/2507.07998/) (2025)
- [Kwai Keye-VL 1.5 Technical Report](https://mlsanity.com/paper/2509.01563/) (2025)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Challenges with unsupervised LLM knowledge discovery](https://mlsanity.com/paper/2312.10029/) (2023)
- [Simple o3: Towards Interleaved Vision-Language Reasoning](https://mlsanity.com/paper/2508.12109/) (2025)
- [VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform](https://mlsanity.com/paper/2504.14904/) (2025)
- [Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning](https://mlsanity.com/paper/2509.22611/) (2025)
- [Efficient End-to-end Language Model Fine-tuning on Graphs](https://mlsanity.com/paper/2312.04737/) (2023)
- [JudgeRLVR: Judge First, Generate Second for Efficient Reasoning](https://mlsanity.com/paper/2601.08468/) (2026)

## Built on by (most cited first, 5 of 5)

- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Kwai Keye-VL-2.0 Technical Report](https://mlsanity.com/paper/2606.10651/) (2026)
- [VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning](https://mlsanity.com/paper/2605.28023/) (2026)
- [Context-Aware RL for Agentic and Multimodal LLMs](https://mlsanity.com/paper/2606.17053/) (2026)
- [FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents](https://mlsanity.com/paper/2610.07898/) (2026)

## Abstract (arXiv)

We propose ContextRL, a novel framework that leverages context augmentation to overcome these bottlenecks. Specifically, to enhance Identifiability, we provide the reward model with full reference solutions as context, enabling fine-grained process verification to filter out false positives (samples with the right answer but low-quality reasoning process). To improve Reachability, we introduce a multi-turn sampling strategy where the reward model generates mistake reports for failed attempts, guiding the policy to "recover" correct responses from previously all-negative groups. Experimental results on 11 perception and reasoning benchmarks show that ContextRL significantly improves knowledge discovery efficiency. Notably, ContextRL enables the Qwen3-VL-8B model to achieve performance comparable to the 32B model, outperforming standard RLVR baselines by a large margin while effectively mitigating reward hacking. Our in-depth analysis reveals the significant potential of contextual information for improving reward model accuracy and document the widespread occurrence of reward hacking, offering valuable insights for future RLVR research.