# Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems

arXiv: https://arxiv.org/abs/2204.07135 · HTML: https://mlsanity.com/paper/2204.07135/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 8 of 8)

- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Off-policy Bandits with Deficient Support](https://mlsanity.com/paper/2006.09438/) (2020)
- [Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents](https://mlsanity.com/paper/2010.11230/) (2020)
- [Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation](https://mlsanity.com/paper/1911.08567/) (2019)
- [A scalable framework for learning from implicit user feedback to improve natural language understanding in large-scale conversational AI systems](https://mlsanity.com/paper/2010.12251/) (2020)
- [Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration](https://mlsanity.com/paper/2103.03373/) (2021)
- [Lessons from Contextual Bandit Learning in a Customer Support Bot](https://mlsanity.com/paper/1905.02219/) (2019)
- [Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents](https://mlsanity.com/paper/2006.07113/) (2020)

## Built on by (most cited first, 4 of 4)

- [Improving Tool Retrieval by Leveraging Large Language Models for Query Generation](https://mlsanity.com/paper/2412.03573/) (2024)
- [Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems](https://mlsanity.com/paper/2209.08429/) (2022)
- [Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems](https://mlsanity.com/paper/2305.10528/) (2023)
- [Data Augmentation for Improving Tail-traffic Robustness in Skill-routing for Dialogue Systems](https://mlsanity.com/paper/2306.04823/) (2023)

## Abstract (arXiv)

Skill routing is an important component in large-scale conversational systems. In contrast to traditional rule-based skill routing, state-of-the-art systems use a model-based approach to enable natural conversations. To provide supervision signal required to train such models, ideas such as human annotation, replication of a rule-based system, relabeling based on user paraphrases, and bandit-based learning were suggested. However, these approaches: (a) do not scale in terms of the number of skills and skill on-boarding, (b) require a very costly expert annotation/rule-design, (c) introduce risks in the user experience with each model update. In this paper, we present a scalable self-learning approach to explore routing alternatives without causing abrupt policy changes that break the user experience, learn from the user interaction, and incrementally improve the routing via frequent model refreshes. To enable such robust frequent model updates, we suggest a simple and effective approach that ensures controlled policy updates for individual domains, followed by an off-policy evaluation for making deployment decisions without any need for lengthy A/B experimentation. We conduct various offline and online A/B experiments on a commercial large-scale conversational system to demonstrate the effectiveness of the proposed method in real-world production settings.