# FACTS About Building Retrieval Augmented Generation-based Chatbots

arXiv: https://arxiv.org/abs/2407.07858 · HTML: https://mlsanity.com/paper/2407.07858/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Improving Retrieval for RAG based Question Answering Models on Financial Documents](https://mlsanity.com/paper/2404.07221/) (2024)

## Built on by (most cited first, 15 of 15)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions](https://mlsanity.com/paper/2507.18910/) (2025)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval](https://mlsanity.com/paper/2502.20969/) (2025)
- [RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice](https://mlsanity.com/paper/2502.14930/) (2025)
- [Generative AI Enhanced Financial Risk Management Information Retrieval](https://mlsanity.com/paper/2504.06293/) (2025)
- [RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines](https://mlsanity.com/paper/2504.13587/) (2025)
- [Development of Application-Specific Large Language Models to Facilitate Research Ethics Review](https://mlsanity.com/paper/2501.10741/) (2025)
- [A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent](https://mlsanity.com/paper/2509.19209/) (2025)
- [Evolutionary Context Search for Automated Skill Acquisition](https://mlsanity.com/paper/2602.16113/) (2026)
- [Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement](https://mlsanity.com/paper/2604.09579/) (2026)
- [Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement](https://mlsanity.com/paper/2510.27051/) (2025)
- [Systematizing LLM Persona Design: A Four-Quadrant Technical Taxonomy for AI Companion Applications](https://mlsanity.com/paper/2511.02979/) (2025)
- [Induced Numerical Instability: Hidden Costs in Multimodal Large Language Models](https://mlsanity.com/paper/2603.04453/) (2026)
- [A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2609.07075/) (2026)

## Abstract (arXiv)

Enterprise chatbots, powered by generative AI, are emerging as key applications to enhance employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and orchestration frameworks like Langchain and Llamaindex are crucial for building these chatbots. However, creating effective enterprise chatbots is challenging and requires meticulous RAG pipeline engineering. This includes fine-tuning embeddings and LLMs, extracting documents from vector databases, rephrasing queries, reranking results, designing prompts, honoring document access controls, providing concise responses, including references, safeguarding personal information, and building orchestration agents. We present a framework for building RAG-based chatbots based on our experience with three NVIDIA chatbots: for IT/HR benefits, financial earnings, and general content. Our contributions are three-fold: introducing the FACTS framework (Freshness, Architectures, Cost, Testing, Security), presenting fifteen RAG pipeline control points, and providing empirical results on accuracy-latency tradeoffs between large and small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots."