# Say What I Want: Towards the Dark Side of Neural Dialogue Models

arXiv: https://arxiv.org/abs/1909.06044 · HTML: https://mlsanity.com/paper/1909.06044/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2019.

## Built on by (most cited first, 6 of 6)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation](https://mlsanity.com/paper/2212.01810/) (2022)
- [Automatically Exposing Problems with Neural Dialog Models](https://mlsanity.com/paper/2109.06950/) (2021)
- [Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs](https://mlsanity.com/paper/2507.16951/) (2025)

## Abstract (arXiv)

Neural dialogue models have been widely adopted in various chatbot applications because of their good performance in simulating and generalizing human conversations. However, there exists a dark side of these models -- due to the vulnerability of neural networks, a neural dialogue model can be manipulated by users to say what they want, which brings in concerns about the security of practical chatbot services. In this work, we investigate whether we can craft inputs that lead a well-trained black-box neural dialogue model to generate targeted outputs. We formulate this as a reinforcement learning (RL) problem and train a Reverse Dialogue Generator which efficiently finds such inputs for targeted outputs. Experiments conducted on a representative neural dialogue model show that our proposed model is able to discover such desired inputs in a considerable portion of cases. Overall, our work reveals this weakness of neural dialogue models and may prompt further researches of developing corresponding solutions to avoid it.