# Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models arXiv: https://arxiv.org/abs/2605.25443 ยท HTML: https://mlsanity.com/paper/2605.25443/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 43 of its 58 references that are in the map. - First posted on arXiv 25 May 2026. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) ## Builds on (most cited first, 30 of 43) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023) - [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017) - [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020) - [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022) - [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023) - [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025) - [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017) - [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018) - [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019) - [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025) - [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021) - [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023) - [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025) - [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026) - [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023) - [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025) - [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026) - [Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models](https://mlsanity.com/paper/2505.23564/) (2025) ## Built on by (most cited first, 2 of 2) - [Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents](https://mlsanity.com/paper/2608.21830/) (2026) - [A helps B while B hurts A: directed transfer in instruction-tuning mixture](https://mlsanity.com/paper/2609.39702/) (2026) ## Abstract (arXiv) Post-training via Reinforcement Learning (RL) has enabled Large Reasoning Models (LRMs) to achieve strong performance in individual domain. However, real-world applications increasingly require general-purpose reasoners rendering strong performance across diverse domains. Mixed-domain post-training aims to achieve this goal by jointly training with mixed domain data, but this often induces capability compromise and degradation among different domains. Existing methods attribute this performance degradation to harmful cross-domain interactions and propose various strategies to mitigate them, but these strategies may also impede beneficial knowledge sharing across domains and in turn fail to match or surpass single-domain performance. To address this problem, we propose Multi-domain Contrastive Policy Optimization (MCPO), which uses contrastive learning to utilize both positive and negative cross-domain interactions for knowledge sharing and competition. Specifically, we partition each rollout generated by LRMs according to its underlying reasoning structures and use reasoning segments to capture these structures. We thus formulate positive and negative pairs of reasoning segments as mutually augmented examples, which provide supportive and competing signals for knowledge sharing. Subsequently, we design complementary contrastive objectives for cross-domain knowledge sharing and intra-domain knowledge consolidation, targeting compatibility across domains and discriminability within each domain to form a harmonious reasoning space. Experimental results across a broad range of domains show that MCPO alleviates performance degradation caused by mixed-domain training and outperforms single-domain training in most cases.