# Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

arXiv: https://arxiv.org/abs/2603.14217 · HTML: https://mlsanity.com/paper/2603.14217/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 43 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 6 of 6)

- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems](https://mlsanity.com/paper/2401.13256/) (2024)
- [Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Persona](https://mlsanity.com/paper/2305.11482/) (2023)
- [DHP Benchmark: Are LLMs Good NLG Evaluators?](https://mlsanity.com/paper/2408.13704/) (2024)
- [MIRACLE: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control](https://mlsanity.com/paper/2310.18342/) (2023)
- [What is a Theory ?](https://mlsanity.com/paper/2305.15780/) (2023)

## Built on by (most cited first, 1 of 1)

- [H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions](https://mlsanity.com/paper/2606.09461/) (2026)

## Abstract (arXiv)

In cognitive science and linguistic theory, dialogue is not seen as a chain of independent utterances but rather as a joint activity sustained by coherence, consistency, and shared understanding. However, many systems for open-domain and personalized dialogue use surface-level similarity metrics (e.g., BLEU, ROUGE, F1) as one of their main reporting measures, which fail to capture these deeper aspects of conversational quality. We re-examine a notable retrieval-augmented framework for personalized dialogue, LAPDOG, as a case study for evaluation methodology. Using both human and LLM-based judges, we identify limitations in current evaluation practices, including corrupted dialogue histories, contradictions between retrieved stories and persona, and incoherent response generation. Our results show that human and LLM judgments align closely but diverge from lexical similarity metrics, underscoring the need for cognitively grounded evaluation methods. Broadly, this work charts a path toward more reliable assessment frameworks for retrieval-augmented dialogue systems that better reflect the principles of natural human communication.