# Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems

arXiv: https://arxiv.org/abs/2305.10528 · HTML: https://mlsanity.com/paper/2305.10528/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 10 of 10)

- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Learning from eXtreme Bandit Feedback](https://mlsanity.com/paper/2009.12947/) (2020)
- [Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents](https://mlsanity.com/paper/2010.11230/) (2020)
- [A scalable framework for learning from implicit user feedback to improve natural language understanding in large-scale conversational AI systems](https://mlsanity.com/paper/2010.12251/) (2020)
- [Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration](https://mlsanity.com/paper/2103.03373/) (2021)
- [Lessons from Contextual Bandit Learning in a Customer Support Bot](https://mlsanity.com/paper/1905.02219/) (2019)
- [Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems](https://mlsanity.com/paper/2204.07135/) (2022)
- [Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems](https://mlsanity.com/paper/2209.08429/) (2022)
- [Domain-Aware Contrastive Knowledge Transfer for Multi-domain Imbalanced Data](https://mlsanity.com/paper/2204.01916/) (2022)

## Abstract (arXiv)

Off-Policy reinforcement learning has been a driving force for the state-of-the-art conversational AIs leading to more natural humanagent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose a method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.