# MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation

arXiv: https://arxiv.org/abs/2409.15240 · HTML: https://mlsanity.com/paper/2409.15240/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [CEM: Commonsense-aware Empathetic Response Generation](https://mlsanity.com/paper/2109.05739/) (2021)
- [Personalized LLM Response Generation with Parameterized Memory Injection](https://mlsanity.com/paper/2404.03565/) (2024)
- ["My agent understands me better": Integrating Dynamic Human-like Memory Recall and Consolidation in LLM-Based Agents](https://mlsanity.com/paper/2404.00573/) (2024)
- [Ziya2: Data-centric Learning is All LLMs Need](https://mlsanity.com/paper/2311.03301/) (2023)

## Built on by (most cited first, 15 of 15)

- [From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents](https://mlsanity.com/paper/2505.19549/) (2025)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues](https://mlsanity.com/paper/2602.01313/) (2026)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support](https://mlsanity.com/paper/2602.01885/) (2026)
- [ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models](https://mlsanity.com/paper/2604.08064/) (2026)
- [RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory](https://mlsanity.com/paper/2608.13334/) (2026)
- [HingeMem: Boundary Guided Long-Term Memory with Query Adaptive Retrieval for Scalable Dialogues](https://mlsanity.com/paper/2604.06845/) (2026)
- [When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents](https://mlsanity.com/paper/2609.03467/) (2026)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall](https://mlsanity.com/paper/2604.26243/) (2026)
- [From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents](https://mlsanity.com/paper/2605.25693/) (2026)
- [ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents](https://mlsanity.com/paper/2605.27240/) (2026)
- [LifeSide: Benchmarking Agents as Lifelong Digital Companions](https://mlsanity.com/paper/2606.04660/) (2026)
- [User-Aware Active Knowledge Acquisition for Emotional Support Dialogue](https://mlsanity.com/paper/2605.29715/) (2026)

## Abstract (arXiv)

Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory injection, emotion support (ES) proficiency, and intimacy, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy.