# Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models

arXiv: https://arxiv.org/abs/2005.13170 · HTML: https://mlsanity.com/paper/2005.13170/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 5 of 5)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation](https://mlsanity.com/paper/2212.01810/) (2022)
- [Automatically Exposing Problems with Neural Dialog Models](https://mlsanity.com/paper/2109.06950/) (2021)
- [Safer Conversational AI as a Source of User Delight](https://mlsanity.com/paper/2304.09865/) (2023)

## Abstract (arXiv)

Recently, neural network based dialogue systems have become ubiquitous in our increasingly digitalized society. However, due to their inherent opaqueness, some recently raised concerns about using neural models are starting to be taken seriously. In fact, intentional or unintentional behaviors could lead to a dialogue system to generate inappropriate responses. Thus, in this paper, we investigate whether we can learn to craft input sentences that result in a black-box neural dialogue model being manipulated into having its outputs contain target words or match target sentences. We propose a reinforcement learning based model that can generate such desired inputs automatically. Extensive experiments on a popular well-trained state-of-the-art neural dialogue model show that our method can successfully seek out desired inputs that lead to the target outputs in a considerable portion of cases. Consequently, our work reveals the potential of neural dialogue models to be manipulated, which inspires and opens the door towards developing strategies to defend them.