mlsanityData to 7 Oct 2026

CPPO: Contrastive Perception Policy Optimization for VLM Agents

arXiv 2601.00501 ↗First posted 1 Jan 2026cs.CVcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision--language models (VLMs). Reliable perception is a core requirement for VLM-based agents that must reason and act in open-ended environments: faulty visual grounding cascades directly into faulty actions, hallucinated tool calls, and unsafe decisions. While reinforcement learning (RL) has significantly improved reasoning in language models, extending these advances to multimodal agents requires improving both perception and reasoning. Prior works address this challenge mainly through explicit perception rewards, which often require extra LLM judges, ground-truth annotations, or forced separation of perception from reasoning. CPPO addresses this limitation in a self-supervised manner by extending the RL objective with a Contrastive Perception Loss (CPL) that provides a direct learning signal for visual grounding. The contrastive objective encourages the model to become more sensitive to input visual information. To apply this signal effectively, CPPO identifies perception tokens using an entropy-shift mechanism in the model's output distributions under perturbed images and applies the contrastive loss selectively to those tokens during training. Experiments show that CPPO surpasses prior methods while avoiding extra models, making training more efficient and scalable, and yielding policies that are better suited to perception-critical agentic tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 43 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

33 of 43 references.

References · 33

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  9. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  10. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  11. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
21 more
  1. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  2. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  3. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  4. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  5. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  6. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  7. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  8. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  9. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  10. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  11. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  12. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  13. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  14. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  15. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  16. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 37
  17. 2024Self-Supervised Visual Preference Alignmentcited by 31
  18. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  19. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  20. 2025Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoningcited by 8
  21. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models
  2. 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
  3. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
  4. 2026Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
  5. 2026Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models
  6. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
  7. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
  8. 2026Visually-Guided Policy Optimization for Multimodal Reasoning