# MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards

arXiv: https://arxiv.org/abs/2601.05488 · HTML: https://mlsanity.com/paper/2601.05488/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Builds on (most cited first, 15 of 15)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [Focused Transformer: Contrastive Training for Context Scaling](https://mlsanity.com/paper/2307.03170/) (2023)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [SCM: Enhancing Large Language Model with Self-Controlled Memory Framework](https://mlsanity.com/paper/2304.13343/) (2023)

## Built on by (most cited first, 13 of 13)

- [MemMA: Coordinating the Memory Cycle through Multi-Agent Reasoning and In-Situ Self-Evolution](https://mlsanity.com/paper/2603.18718/) (2026)
- [MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs](https://mlsanity.com/paper/2605.08374/) (2026)
- [AttriMem: Attribution-Guided Process Feedback for Agent Memory Construction](https://mlsanity.com/paper/2607.21106/) (2026)
- [Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory](https://mlsanity.com/paper/2605.19952/) (2026)
- [MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents](https://mlsanity.com/paper/2607.24097/) (2026)
- [APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering](https://mlsanity.com/paper/2609.02253/) (2026)
- [CMI-Mem: Toward Generalizable Long-Term Memory Management via CMI-Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.20553/) (2026)
- [Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner](https://mlsanity.com/paper/2608.23268/) (2026)
- [Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit](https://mlsanity.com/paper/2608.29605/) (2026)
- [Memory Control Signals Emerge Before Action in Long Horizon Agents](https://mlsanity.com/paper/2609.27286/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillation](https://mlsanity.com/paper/2609.39740/) (2026)
- [MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories](https://mlsanity.com/paper/2609.40195/) (2026)

## Abstract (arXiv)

Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states. While memory-augmented frameworks offer a structured alternative, current systems rely on static prompting of closed-source models or suffer from ineffective training paradigms with sparse rewards. We introduce MemBuilder, a reinforcement learning framework that trains models to orchestrate multi-dimensional memory construction with attributed dense rewards. MemBuilder addresses two key challenges: (1) Sparse Trajectory-Level Rewards: we employ synthetic session-level question generation to provide dense intermediate rewards across extended trajectories; and (2) Multi-Dimensional Memory Attribution: we introduce contribution-aware gradient weighting that scales policy updates based on each component's downstream impact. Experimental results show that MemBuilder enables a 4B-parameter model to outperform state-of-the-art closed-source baselines, exhibiting strong generalization across long-term dialogue benchmarks.