mlsanityData to 7 Oct 2026

Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

arXiv 2606.08867 ↗First posted 7 Jun 2026cs.CLcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The rapid rise in LLM capabilities has made AI agents increasingly viable across a broad range of tasks. Among the most promising applications is building production-ready customer-facing agents, a challenge that demands coordinated excellence in evaluation methodology, context engineering, training, and online measurement. Yet these critical pillars are typically developed in isolation, creating blind spots that only surface after deployment. In this paper, we present a unified framework that bridges offline development with online impact for customer support AI agents at Nubank, a company with 100M+ users. Our approach integrates several key components: (1) structured context engineering tailored to customer support agents, (2) systematic human-in-the-loop prompt iteration, (3) rigorous LLM judge evaluation with measured inter-rater agreement and GEPA optimization for consistency, and (4) ideation-to-production validation. A central insight is that evaluation-pipeline quality directly determines iteration velocity. We present results from five production deployments spanning distinct domains: card delivery, debt management, credit-limit support, card management, and product explanation. These deployments deliver consistent customer-satisfaction gains while substantially accelerating iteration. In our card-delivery deployment, large-scale A/B testing yields a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate over prior agent variants, alongside a strong correlation between offline simulation metrics and online outcomes, demonstrating that eval-driven development reliably predicts production impact. On most use cases, AI satisfaction reaches within a few percentage points of expert human agents.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 54 references that are in the map. First posted on arXiv 7 Jun 2026.

Builds on

26 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  2. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks12 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2023Large Language Models as Optimizerscited by 830
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  12. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
12 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  4. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  5. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  6. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
  7. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  8. 2025LLM-Friendly Knowledge Representation for Customer Supportcited by 11
  9. 2023Schema-Guided User Satisfaction Modeling for Task-Oriented Dialoguescited by 8
  10. 2023Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Supportcited by 6
  11. 2023NatCS: Eliciting Natural Customer Support Dialoguescited by 5
  12. 2023Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistantscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scalecited by 1
  2. 2026Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scalecited by 0
  3. 2026Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support
  2. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Construction
  3. 2026Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
  4. 2025Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
  5. 2026AlphaEval: Evaluating Agents in Production
  6. 2025UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI
  7. 2026Helping Customers in Distress: An LLM-powered Agent that Converses, Probes, and Routes
  8. 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents