mlsanityData to 14 Oct 2026

A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Tasks

arXiv 2510.11897 ↗First posted 13 Oct 2025cs.HCcited by 0

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Grounding conversations in existing passages, known as Retrieval-Augmented Generation (RAG), is an important aspect of Chat-Based Assistants powered by Large Language Models (LLMs) to ensure they are faithful and don't provide misinformation. Several benchmarks have been created to measure the performance of LLMs on this task. We present a longitudinal study comparing the feedback loop of an internal and external human annotator group for the complex annotation task of creating multi-turn RAG conversations for evaluating LLMs. We analyze the conversations produced by both groups and provide results of a survey comparing their experiences. Our study highlights the advantages of each annotator population and the impact of the different feedback loops; a closer loop creates higher quality conversations with a decrease in quantity and diversity. Further, we present guidance for how to best utilize two different population groups when performing annotation tasks, particularly when the task is complex.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 23 of its 77 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

23 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits22 Aug
  2. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems7 Jan
  3. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues19 Sep

Other references · 20

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2018CoQA: A Conversational Question Answering Challengecited by 683
  3. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  4. 2018QuAC : Question Answering in Contextcited by 381
  5. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  6. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 148
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  8. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 109
  9. 2021TopiOCQA: Open-domain Conversational Question Answering with Topic Switchingcited by 96
  10. 2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53
  11. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 50
  12. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
8 more
  1. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  2. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  3. 2016Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revisioncited by 13
  4. 2024Retrieval-Augmented Data Augmentation for Low-Resource Domain Taskscited by 12
  5. 2021Conversational Entity Linking: Problem Definition and Datasetscited by 11
  6. 2021Generative Conversational Networkscited by 11
  7. 2021A Survey on Task Assignment in Crowdsourcingcited by 8
  8. 2021Understanding How Programmers Can Use Annotations on Documentationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  2. 2025Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables
  3. 2026Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG
  4. 2024Adaptive Retrieval-Augmented Generation for Conversational Systems
  5. 2024A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
  6. 2026RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation
  7. 2026Comprehensive Comparison of RAG Methods Across Multi-Domain Conversational QA
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models