mlsanityData to 7 Oct 2026

Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

arXiv 2607.18039 ↗First posted 20 Jul 2026cs.IRcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an Evidence-Grounded Customer-Service Agent Workflow deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~citelanggraph2024. The paper contributes three reusable deployment patterns: hybrid RAG evidence construction, where multi-channel retrieval and reranking produce auditable FAQ candidates; evidence-grounded issue/action decision, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and trace-driven RAG and reranker improvement, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 13 of its 27 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

13 of 27 references.

References · 13

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2019Passage Re-ranking with BERTcited by 629
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  10. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  11. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  12. 2022Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Modelscited by 26
1 more
  1. 2024DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automatoncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support
  2. 2026AgentSim: A Platform for Verifiable Agent-Trace Simulation
  3. 2026MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
  4. 2026Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG
  5. 2026The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commerce
  6. 2026FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments
  7. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification
  8. 2026Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation