# MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

arXiv: https://arxiv.org/abs/2602.23184 · HTML: https://mlsanity.com/paper/2602.23184/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 6 of 6)

- [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [Understanding User Experience in Large Language Model Interactions](https://mlsanity.com/paper/2401.08329/) (2024)
- [CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems](https://mlsanity.com/paper/2404.02103/) (2024)
- [RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues](https://mlsanity.com/paper/2409.12558/) (2024)
- [TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants](https://mlsanity.com/paper/2405.02637/) (2024)

## Built on by (most cited first, 9 of 9)

- [RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation](https://mlsanity.com/paper/2605.04523/) (2026)
- [H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversations](https://mlsanity.com/paper/2605.00631/) (2026)
- [AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations](https://mlsanity.com/paper/2603.10524/) (2026)
- [Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking](https://mlsanity.com/paper/2605.12028/) (2026)
- [uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking](https://mlsanity.com/paper/2606.11945/) (2026)
- [Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG](https://mlsanity.com/paper/2606.28352/) (2026)
- [5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control](https://mlsanity.com/paper/2606.28737/) (2026)
- [Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation](https://mlsanity.com/paper/2609.35308/) (2026)
- [Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG](https://mlsanity.com/paper/2609.36700/) (2026)

## Abstract (arXiv)

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark