AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
On the map: Retrieval-augmented generation → RAG pipeline design and evaluation also fits: Legal and financial RAG
Abstract (arXiv)
Enterprises adopting retrieval-augmented generation (RAG) face a recurring operational decision: promote, revise, or block a system version. The evidence is incomplete and the metrics come from fallible LLM judges. We report on AGO AI Quality Gate (AGO), an evidence-first quality-gate framework deployed in industrial RAG assessment engagements. AGO integrates four key components: a four-state decision model that treats missing data and judge errors as explicit outcomes; layered scoring combining deterministic checks, local guardrails, and structured LLM evaluation; a stratified beta-binomial gate that quantifies regression risk probabilistically; and a mandatory meta-evaluation protocol to validate the LLM judge before it influences decisions. Since engagement data is proprietary, we evaluate the judge layer on RAGBench, a public benchmark of 100k annotated RAG traces across 12 datasets. On identical stratified test samples (N=1200 per judge), a low-cost judge (gpt-4.1-nano) detects non-adherent answers barely above chance (AUROC 0.603 [0.570, 0.634]), despite producing flawless protocol output, while gpt-4o reaches 0.783 [0.756, 0.807] -- yet its per-domain performance still ranges from 0.62 to 0.88. A fixed-seed gate study spanning regression, no change, and improvement quantifies unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile reduces unsafe promotion to 22.2%-35.1%, against 29.3%-41.8% for a naive gate. These results support the design choices that judge quality must be measured per engagement and that point estimates alone are not a release decision.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 12 of its 17 references that are in the map. First posted on arXiv 1 Oct 2026.
Builds on
12 of 17 references.
Closest ancestors · 4
Same direction, newest first
- 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
- 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
- 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep
Other references · 8
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
- 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
- 2025Towards a rigorous evaluation of RAG systems: the challenge of due diligence
- 2026A Fixed-Budget, Cluster-Aware Standard for LLM-as-a-Judge Evaluation: A Multi-Hop RAG Stress Test
- 2025Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance
- 2025Data Quality Challenges in Retrieval-Augmented Generation
- 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets
- 2026The RAT: A Unified Bayesian Model for RAG Evaluation
- 2026Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys