mlsanityData to 14 Oct 2026

Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support

arXiv 2510.06674 ↗First posted 8 Oct 2025cs.AIcited by 10

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 9 of its 12 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

9 of 12 references.

References · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  7. 2025Reinforcement Learning from Human Feedbackcited by 58
  8. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  9. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Supportcited by 48
  2. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  3. 2026SEAR: Schema-Based Evaluation and Routing for LLM Gatewayscited by 1
  4. 2026Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commercecited by 1
  5. 2026Demand-Driven Context: A Methodology for Building Enterprise Knowledge Bases Through Agent Failurecited by 0
  6. 2026FRAMES: Guarded and Dual-Objective Skill Evolution for Agents in Policy-Governed Enterprise Workflowscited by 0
  7. 2026SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedbackcited by 0
  8. 2026From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scalecited by 0
  9. 2026RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agentscited by 0
  10. 2026State-Grounded Conditioning: Wrapping User-Facing LLM Agents Where Direction Depends on Live Statecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework
  2. 2026Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
  3. 2026Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
  4. 2025MindFlow: Revolutionizing E-commerce Customer Support with Multimodal LLM Agents
  5. 2026Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents
  6. 2026Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback
  7. 2025MemOrb: A Plug-and-Play Verbal-Reinforcement Memory Layer for E-Commerce Customer Service
  8. 2026Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support