# Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues

arXiv: https://arxiv.org/abs/2602.01313 · HTML: https://mlsanity.com/paper/2602.01313/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory](https://mlsanity.com/paper/2512.06688/) (2025)
- [MemInsight: Autonomous Memory Augmentation for LLM Agents](https://mlsanity.com/paper/2503.21760/) (2025)
- [Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models](https://mlsanity.com/paper/2308.15022/) (2023)
- [Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs](https://mlsanity.com/paper/2510.27246/) (2025)
- [PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants](https://mlsanity.com/paper/2506.09902/) (2025)
- [Needle in the Haystack for Memory Based Large Language Models](https://mlsanity.com/paper/2407.01437/) (2024)
- [Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents](https://mlsanity.com/paper/2411.16740/) (2024)
- [MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation](https://mlsanity.com/paper/2409.15240/) (2024)
- [ProCIS: A Benchmark for Proactive Retrieval in Conversations](https://mlsanity.com/paper/2405.06460/) (2024)
- [Interpersonal Memory Matters: A New Task for Proactive Dialogue Utilizing Conversational History](https://mlsanity.com/paper/2503.05150/) (2025)
- [Current and Future Use of Large Language Models for Knowledge Work](https://mlsanity.com/paper/2503.16774/) (2025)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration](https://mlsanity.com/paper/2512.02981/) (2025)
- [Advancing Multi-Party Dialogue Framework with Speaker-ware Contrastive Learning](https://mlsanity.com/paper/2501.11292/) (2025)

## Built on by (most cited first, 22 of 22)

- [GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents](https://mlsanity.com/paper/2606.18829/) (2026)
- [MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations](https://mlsanity.com/paper/2607.12893/) (2026)
- [H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions](https://mlsanity.com/paper/2606.09461/) (2026)
- [MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents](https://mlsanity.com/paper/2603.05697/) (2026)
- [RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory](https://mlsanity.com/paper/2608.13334/) (2026)
- [AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents](https://mlsanity.com/paper/2606.21144/) (2026)
- [SocialMemBench: Are AI Memory Systems Ready for Social Group Settings?](https://mlsanity.com/paper/2605.17789/) (2026)
- [FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents](https://mlsanity.com/paper/2606.31522/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [ArborMem: Navigating Interaction States with Memory Forests](https://mlsanity.com/paper/2608.17534/) (2026)
- [MemFuse: Multi-Source Memory Fusion from Fragmented Observations](https://mlsanity.com/paper/2608.18704/) (2026)
- [AutoViewMem: Self-Configuring Orthogonal Views for Conversational Long-Term Memory](https://mlsanity.com/paper/2609.21940/) (2026)
- [Propose, Verify, Commit: Evidence-Grounded Memory for Long-Horizon Multi-Actor Conversations](https://mlsanity.com/paper/2609.23465/) (2026)
- [Probing Stability-Plasticity Tradeoffs in Agent Memory through Cognitive Experimental Paradigms](https://mlsanity.com/paper/2609.30558/) (2026)
- [Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations](https://mlsanity.com/paper/2609.32522/) (2026)
- [Audience-Bound Persistent Memory: Authorization Across the Memory Lifecycle](https://mlsanity.com/paper/2609.36373/) (2026)
- [AgentMemGate: Addressing Speculation Contamination in Conversational Assistant Memory](https://mlsanity.com/paper/2610.07707/) (2026)
- [PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue](https://mlsanity.com/paper/2610.07725/) (2026)
- [DSV-Mem: Evaluating Multimodal Memory in Professional Workflows for MLLM Agents](https://mlsanity.com/paper/2610.08102/) (2026)
- [Memory Depth and Reconstructed Context Width: A Controlled Evaluation of Hierarchical Retrieval](https://mlsanity.com/paper/2610.08300/) (2026)
- [Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory](https://mlsanity.com/paper/2610.12874/) (2026)
- [EGCForest: Structured Agent Memory Organization for Enterprise Group Chats](https://mlsanity.com/paper/2610.19836/) (2026)

## Abstract (arXiv)

Long-term conversational memory in practical LLM applications is inherently collaborative: information is produced by multiple participants, scattered across groups and channels, revised over time, and implicitly grounded in roles and social context. Yet there is currently no established benchmark that evaluates memory under interaction patterns resembling real-world deployment, as existing benchmarks largely focus on dyadic or single-topic dialogues. In this paper, we introduce EverMemBench, the first benchmark designed for long-horizon collaborative memory, built from multi-party, multi-group conversations spanning over one million tokens with dense cross-topic interleaving, temporally evolving decisions, and role-conditioned personas. EverMemBench evaluates memory systems using 2400 QA pairs across three dimensions essential for real applications: fine-grained recall, memory awareness, and user profile understanding. Our evaluation reveals fundamental limitations of current systems: multi-hop reasoning collapses under multi-party attribution even with oracle evidence (26% accuracy), temporal reasoning fails without explicit version semantics beyond timestamps, and memory awareness is bottlenecked by retrieval, as similarity-based methods miss implicitly relevant information. EverMemBench thus represents a concrete step toward realistic evaluation of LLM memory and a cornerstone benchmark for developing next-generation LLMs that reason over time, roles, and collaborative interaction structure. Our benchmark and code are publicly available at https://github.com/EverMind-AI/EverMemBench.