# Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration

arXiv: https://arxiv.org/abs/2103.03373 · HTML: https://mlsanity.com/paper/2103.03373/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Improving Robustness of Task Oriented Dialog Systems](https://mlsanity.com/paper/1911.05153/) (2019)

## Built on by (most cited first, 8 of 8)

- [Large-scale Robustness Analysis of Video Action Recognition Models](https://mlsanity.com/paper/2207.01398/) (2022)
- [Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems](https://mlsanity.com/paper/2204.07135/) (2022)
- [Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems](https://mlsanity.com/paper/2209.08429/) (2022)
- [Handling Long-Tail Queries with Slice-Aware Conversational Systems](https://mlsanity.com/paper/2104.13216/) (2021)
- [Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings](https://mlsanity.com/paper/2305.14299/) (2023)
- [Learning Slice-Aware Representations with Mixture of Attentions](https://mlsanity.com/paper/2106.02363/) (2021)
- [Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems](https://mlsanity.com/paper/2305.10528/) (2023)
- [Data Augmentation for Improving Tail-traffic Robustness in Skill-routing for Dialogue Systems](https://mlsanity.com/paper/2306.04823/) (2023)

## Abstract (arXiv)

Current state-of-the-art large-scale conversational AI or intelligent digital assistant systems in industry comprises a set of components such as Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). For some of these systems that leverage a shared NLU ontology (e.g., a centralized intent/slot schema), there exists a separate skill routing component to correctly route a request to an appropriate skill, which is either a first-party or third-party application that actually executes on a user request. The skill routing component is needed as there are thousands of skills that can either subscribe to the same intent and/or subscribe to an intent under specific contextual conditions (e.g., device has a screen). Ensuring model robustness or resilience in the skill routing component is an important problem since skills may dynamically change their subscription in the ontology after the skill routing model has been deployed to production. We show how different modeling design choices impact the model robustness in the context of skill routing on a state-of-the-art commercial conversational AI system, specifically on the choices around data augmentation, model architecture, and optimization method. We show that applying data augmentation can be a very effective and practical way to drastically improve model robustness.