# Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents

arXiv: https://arxiv.org/abs/2605.21768 · HTML: https://mlsanity.com/paper/2605.21768/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 16 of 16)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems](https://mlsanity.com/paper/2506.07398/) (2025)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models](https://mlsanity.com/paper/2505.22101/) (2025)
- [CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension](https://mlsanity.com/paper/2510.05520/) (2025)

## Built on by (most cited first, 10 of 10)

- [TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents](https://mlsanity.com/paper/2607.13988/) (2026)
- [Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence](https://mlsanity.com/paper/2608.12743/) (2026)
- [The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.06663/) (2026)
- [Rosetta Memory: Adaptive Memory for Cross-LLM Agents](https://mlsanity.com/paper/2606.07711/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue](https://mlsanity.com/paper/2609.26780/) (2026)
- [Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning](https://mlsanity.com/paper/2609.34422/) (2026)
- [Learning What to Remember: Long-horizon Counterfactual Memory Optimization](https://mlsanity.com/paper/2609.37930/) (2026)
- [LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillation](https://mlsanity.com/paper/2609.39740/) (2026)
- [LongHarness: Advancing Agentic Reasoning over 100M-Token Corpora via Stateful Evidence Management](https://mlsanity.com/paper/2610.13075/) (2026)

## Abstract (arXiv)

Memory-augmented LLM agents enable interactions that extend beyond finite context windows by storing, updating, and reusing information across sessions. However, training such agents with reinforcement learning in multi-session environments is challenging because memory turns the agent's past actions into part of its future environment. Once different rollouts write, update, or delete different memories, they no longer share the same intermediate memory state, making trajectory-level comparisons fundamentally unfair. This violates a key assumption behind group-relative methods such as GRPO, where rollouts are compared as if they were sampled from the same effective environment. Consequently, trajectory-level rewards provide noisy or biased credit signals for long-horizon memory operations. To address this challenge, we introduce Memory-R2, a training framework for long-horizon memory-augmented LLM agents. Its core algorithm, LoGo-GRPO, combines local and global group-relative optimization. The global objective preserves end-to-end learning from long-horizon trajectory-level rewards, while local rerollouts compare different memory-operation outcomes from the same intermediate memory state, yielding fairer group comparisons and more precise supervision for memory construction. Beyond credit assignment, Memory-R2 jointly optimizes memory formation and memory evolution with a shared-parameter co-learning design, where a fact extractor and a memory manager are instantiated from the same LLM backbone through role-specific prompts. To stabilize multi-step RL over long memory horizons, we adopt a progressive curriculum that increases the training horizon from 8 to 16 to 32 sessions. Together, these components provide an effective training paradigm for memory-augmented LLM agents in long-horizon multi-session settings.