# Behavior Knowledge Merge in Reinforced Agentic Models

arXiv: https://arxiv.org/abs/2601.13572 · HTML: https://mlsanity.com/paper/2601.13572/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 38)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [Towards Safer Large Language Models through Machine Unlearning](https://mlsanity.com/paper/2402.10058/) (2024)
- [Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors](https://mlsanity.com/paper/2503.08099/) (2025)
- [CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers](https://mlsanity.com/paper/2305.17455/) (2023)
- [LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement](https://mlsanity.com/paper/2504.16053/) (2025)
- [Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning](https://mlsanity.com/paper/2504.09772/) (2025)
- [SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs](https://mlsanity.com/paper/2510.05069/) (2025)
- [Superficial Self-Improved Reasoners Benefit from Model Merging](https://mlsanity.com/paper/2503.02103/) (2025)

## Built on by (most cited first, 11 of 11)

- [CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics](https://mlsanity.com/paper/2605.09584/) (2026)
- [DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation](https://mlsanity.com/paper/2605.09188/) (2026)
- [Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging](https://mlsanity.com/paper/2606.18521/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning](https://mlsanity.com/paper/2605.20075/) (2026)
- [ResMerge: Residual-based Spectral Merging of Large Language Models](https://mlsanity.com/paper/2606.02252/) (2026)
- [Exploring Information Seeking Agent Consolidation](https://mlsanity.com/paper/2602.00585/) (2026)
- [When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis](https://mlsanity.com/paper/2607.16062/) (2026)
- [The evolution of sex for artificial intelligence: a population-genetic framework for multigenerational model populations](https://mlsanity.com/paper/2609.18560/) (2026)
- [Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs](https://mlsanity.com/paper/2609.32821/) (2026)
- [BACAM: Behavior-Aware Continual Agent Merging for Multi-Turn Interaction](https://mlsanity.com/paper/2610.04966/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.