mlsanityData to 7 Oct 2026

Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

arXiv 2407.05925 ↗First posted 8 Jul 2024cs.CL cs.AIcited by 10

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 37 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

20 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  10. 2014Deep metric learning using Triplet networkcited by 421
  11. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
5 more
  1. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  2. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  3. 2017Rasa: Open Source Language Understanding and Dialogue Managementcited by 67
  4. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  5. 2023When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasetscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluationcited by 8
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  3. 2024STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBackcited by 4
  4. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  5. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  6. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategiescited by 1
  7. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Datacited by 0
  8. 2025JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industrycited by 0
  9. 2025HybridRAG: A Practical LLM-based ChatBot Framework based on Pre-Generated Q&A over Raw Unstructured Documentscited by 0
  10. 2026From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering
  2. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  3. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  4. 2025AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  6. 2025Advancing Academic Chatbots: Evaluation of Non Traditional Outputs
  7. 2023A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest
  8. 2024Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?