# AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents

arXiv: https://arxiv.org/abs/2510.08149 · HTML: https://mlsanity.com/paper/2510.08149/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering](https://mlsanity.com/paper/2404.17723/) (2024)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective](https://mlsanity.com/paper/2310.19233/) (2023)
- [Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?](https://mlsanity.com/paper/2402.00841/) (2024)
- [Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge](https://mlsanity.com/paper/2506.00777/) (2025)
- [Natural Language Processing for Human Resources: A Survey](https://mlsanity.com/paper/2410.16498/) (2024)
- [Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization](https://mlsanity.com/paper/2403.00067/) (2024)
- [Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations](https://mlsanity.com/paper/2410.10136/) (2024)
- [AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching](https://mlsanity.com/paper/2305.17619/) (2023)
- [An Effective, Performant Named Entity Recognition System for Noisy Business Telephone Conversation Transcripts](https://mlsanity.com/paper/2209.13736/) (2022)
- [Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop](https://mlsanity.com/paper/2211.01354/) (2022)
- [DialogQAE: N-to-N Question Answer Pair Extraction from Customer Service Chatlog](https://mlsanity.com/paper/2212.07112/) (2022)
- [DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization](https://mlsanity.com/paper/2510.05858/) (2025)
- [Can Post-Training Quantization Benefit from an Additional QLoRA Integration?](https://mlsanity.com/paper/2502.10202/) (2025)

## Built on by (most cited first, 5 of 5)

- [From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents](https://mlsanity.com/paper/2605.15104/) (2026)
- [Beyond Text-to-SQL: An Agentic LLM System for Governed Enterprise Analytics APIs](https://mlsanity.com/paper/2605.21027/) (2026)
- [When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success](https://mlsanity.com/paper/2609.21187/) (2026)
- [Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMs](https://mlsanity.com/paper/2609.27262/) (2026)
- [Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills](https://mlsanity.com/paper/2610.14346/) (2026)

## Abstract (arXiv)

The utilization of conversational AI systems by leveraging Retrieval Augmented Generation (RAG) techniques to solve customer problems has been on the rise with the rapid progress of Large Language Models (LLMs). However, the absence of a company-specific dedicated knowledge base is a major barrier to the integration of conversational AI systems in contact centers. To this end, we introduce AI Knowledge Assist, a system that extracts knowledge in the form of question-answer (QA) pairs from historical customer-agent conversations to automatically build a knowledge base. Fine-tuning a lightweight LLM on internal data demonstrates state-of-the-art performance, outperforming larger closed-source LLMs. More specifically, empirical evaluation on 20 companies demonstrates that the proposed AI Knowledge Assist system that leverages the LLaMA-3.1-8B model eliminates the cold-start gap in contact centers by achieving above 90% accuracy in answering information-seeking questions. This enables immediate deployment of RAG-powered chatbots.