mlsanityData to 14 Oct 2026

MAPO: Mixed Advantage Policy Optimization

arXiv 2509.18849 ↗First posted 23 Sep 2025cs.AIcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reinforcement learning for foundation models, such as Group Relative Policy Optimization (GRPO), have significantly improved the performance of foundation models on reasoning tasks. Notably, the advantage function serves as a central mechanism in GRPO for ranking the trajectory importance. However, existing explorations encounter both advantage reversion and advantage mirror problems, which hinder the reasonable advantage allocation across different query samples. In this work, we propose an easy but effective GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the trajectory appears with different certainty and propose the advantage percent deviation for samples with high-certainty trajectories. Furthermore, we dynamically reweight the advantage function for samples with varying trajectory certainty, thereby adaptively configuring the advantage function to account for sample-specific characteristics. Comparison with related state-of-the-art methods, along with ablation studies on different advantage variants, validates the effectiveness of our approach.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 68 of its 74 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

68 of 74 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Single-stream Policy Optimization16 Sep
  2. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets15 Aug
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  4. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  5. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models5 Jun
  6. 2025Stable Reinforcement Learning for Efficient Reasoning23 May
  7. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward22 May
  8. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  9. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  10. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning12 May
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
11 more
  1. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPO31 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2025Qwen3 Technical Reportcited by 8,820
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2025Qwen2.5-VL Technical Reportcited by 5,587
33 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  13. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  14. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  15. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  16. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  17. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  18. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  19. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  20. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  21. 2025Qwen2.5-1M Technical Reportcited by 164
  22. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  23. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  24. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 82
  25. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  26. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  27. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  28. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 47
  29. 2018Contemplating Visual Emotions: Understanding and Overcoming Dataset Biascited by 41
  30. 2025A Survey of Direct Preference Optimizationcited by 36
  31. 2024VILA2: VILA Augmented VILAcited by 25
  32. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  33. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  2. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  3. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  4. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  5. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
  6. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  7. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Modelscited by 5
  8. 2026OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Taskscited by 2
  9. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Followingcited by 2
  10. 2026Can the Environment Speak for Itself? T2-GRPO: A Turn-Trajectory Group Relative Policy Optimization for Caregiver Agentscited by 0
  11. 2026Spurious Advantage Hidden in GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
  2. 2026GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization with Enhanced Training Data Utilization for Sample-Constrained Reinforcement Learning
  3. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  4. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
  7. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  8. 2026LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models