# Feedback-Based Self-Learning in Large-Scale Conversational AI Agents

arXiv: https://arxiv.org/abs/1911.02557 · HTML: https://mlsanity.com/paper/1911.02557/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2105.04850/) (2021)
- [Correcting Automated and Manual Speech Transcription Errors using Warped Language Models](https://mlsanity.com/paper/2103.14580/) (2021)
- [Robust Training for Conversational Question Answering Models with Reinforced Reformulation Generation](https://mlsanity.com/paper/2310.13505/) (2023)
- [Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI](https://mlsanity.com/paper/2501.06129/) (2025)
- [Mondegreen: A Post-Processing Solution to Speech Recognition Error Correction for Voice Search Queries](https://mlsanity.com/paper/2105.09930/) (2021)
- [Self-Aware Feedback-Based Self-Learning in Large-Scale Conversational AI](https://mlsanity.com/paper/2205.00029/) (2022)
- [PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding](https://mlsanity.com/paper/2210.12308/) (2022)
- [A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS](https://mlsanity.com/paper/2212.03817/) (2022)
- [Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models](https://mlsanity.com/paper/2109.01754/) (2021)
- [Unraveling Code-Mixing Patterns in Migration Discourse: Automated Detection and Analysis of Online Conversations on Reddit](https://mlsanity.com/paper/2406.08633/) (2024)
- [Spelling Correction through Rewriting of Non-Autoregressive ASR Lattices](https://mlsanity.com/paper/2409.16469/) (2024)
- [Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Prediction](https://mlsanity.com/paper/2505.18731/) (2025)
- [Emergence of Agentic AI: A Review on Evolution, Background, Working Principles, Applications, Adoption Factors, and Future Research Directions](https://mlsanity.com/paper/2608.18110/) (2026)
- [Federated Agent Optimization](https://mlsanity.com/paper/2610.01195/) (2026)

## Abstract (arXiv)

Today, most large-scale conversational AI agents (e.g. Alexa, Siri, or Google Assistant) are built using manually annotated data to train the different components of the system. Typically, the accuracy of the ML models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages user-system interaction feedback signals to automate learning without any manual annotation. Users here tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win/loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.