mlsanityData to 7 Oct 2026

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

arXiv 2603.10101 ↗First posted 10 Mar 2026cs.LG cs.AI cs.CLcited by 5

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 67 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

59 of 67 references.

Closest ancestors · 23

Same direction, newest first

  1. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs16 Jan
  2. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward18 Dec
  3. 2025Soft Adaptive Policy Optimization25 Nov
  4. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning30 Oct
  5. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences8 Oct
  6. 2025Rethinking Entropy Regularization in Large Reasoning Models29 Sep
  7. 2025ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search27 Aug
  8. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  9. 2025Geometric-Mean Policy Optimization28 Jul
  10. 2025Agentic Reinforced Policy Optimization26 Jul
  11. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
11 more
  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  7. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  8. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  10. 2023Let's Verify Step by Step31 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2020Supervised Contrastive Learningcited by 2,803
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
24 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2025Group Sequence Policy Optimizationcited by 788
  6. 2019On Variational Bounds of Mutual Informationcited by 550
  7. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  8. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  9. 2020Understanding the Behaviour of Contrastive Losscited by 397
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  11. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  12. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  13. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  14. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  15. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  16. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  17. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  18. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  19. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  20. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 37
  21. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  22. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14
  23. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 10
  24. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  2. 2026Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Modelscited by 2
  3. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradientscited by 1
  4. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspectivecited by 1
  5. 2026Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  2. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  3. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  4. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
  5. 2026Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
  6. 2026Linear Dynamics in the RLVR Training of Large Language Models
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2026Contrastive Reinforced Policy Optimization via Privileged Self-Distillation