mlsanityData to 13 Oct 2026

From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants

arXiv 2602.15859 ↗First posted 26 Jan 2026cs.CLcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Building reliable conversational AI assistants for customer-facing industries remains challenging due to noisy conversational data, fragmented knowledge, and the requirement for accurate human hand-off - particularly in domains that depend heavily on real-time information. This paper presents an end-to-end framework for constructing and evaluating a conversational AI assistant directly from historical call transcripts. Incoming transcripts are first graded using a simplified adaptation of the PIPA framework, focusing on observation alignment and appropriate response behavior, and are filtered to retain only high-quality interactions exhibiting coherent flow and effective human agent responses. Structured knowledge is then extracted from curated transcripts using large language models (LLMs) and deployed as the sole grounding source in a Retrieval-Augmented Generation (RAG) pipeline. Assistant behavior is governed through systematic prompt tuning, progressing from monolithic prompts to lean, modular, and governed designs that ensure consistency, safety, and controllable execution. Evaluation is conducted using a transcript-grounded user simulator, enabling quantitative measurement of call coverage, factual accuracy, and human escalation behavior. Additional red teaming assesses robustness against prompt injection, out-of-scope, and out-of-context attacks. Experiments are conducted in the Real Estate and Specialist Recruitment domains, which are intentionally challenging and currently suboptimal for automation due to their reliance on real-time data. Despite these constraints, the assistant autonomously handles approximately 30 percents of calls, achieves near-perfect factual accuracy and rejection behavior, and demonstrates strong robustness under adversarial testing.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 41 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

23 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Cloning a Conversational Voice AI Agent from Call,Recording Datasets for Telesales5 Sep
  2. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges15 May
  3. 2025A Survey of AI Agent Protocols23 Apr

Other references · 20

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
8 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,093
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 877
  4. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 359
  5. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 120
  6. 2021The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systemscited by 39
  7. 2022Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systemscited by 28
  8. 2018Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Timecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents
  2. 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
  3. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents
  4. 2026Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
  5. 2025KnowsLM: A framework for evaluation of small language models for knowledge augmentation and humanised conversations
  6. 2025Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation
  7. 2026On Evaluating and Improving Conversational Agents in Production
  8. 2025Conversational Alignment with Artificial Intelligence in Context