# The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identity

arXiv: https://arxiv.org/abs/2106.02692 · HTML: https://mlsanity.com/paper/2106.02692/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Ethical Challenges in Data-Driven Dialogue Systems](https://mlsanity.com/paper/1711.09050/) (2017)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations](https://mlsanity.com/paper/2008.12348/) (2020)
- [LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing](https://mlsanity.com/paper/2105.01992/) (2021)

## Built on by (most cited first, 16 of 16)

- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Mirages: On Anthropomorphism in Dialogue Systems](https://mlsanity.com/paper/2305.09800/) (2023)
- [NLP Verification: Towards a General Methodology for Certifying Robustness](https://mlsanity.com/paper/2403.10144/) (2024)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Thinking beyond the anthropomorphic paradigm benefits LLM research](https://mlsanity.com/paper/2502.09192/) (2025)
- [A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies](https://mlsanity.com/paper/2502.09870/) (2025)
- [Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems](https://mlsanity.com/paper/2502.14019/) (2025)
- [Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems](https://mlsanity.com/paper/2210.12429/) (2022)
- [ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification](https://mlsanity.com/paper/2305.04003/) (2023)
- [The State of Profanity Obfuscation in Natural Language Processing](https://mlsanity.com/paper/2210.07595/) (2022)
- [From Pixels to Personas: Investigating and Modeling Self-Anthropomorphism in Human-Robot Dialogues](https://mlsanity.com/paper/2410.03870/) (2024)
- [Disclosure By Design: Identity Transparency as a Behavioural Property of Conversational AI Models](https://mlsanity.com/paper/2603.16874/) (2026)
- [A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception](https://mlsanity.com/paper/2208.05757/) (2022)
- [RealityTest: How People Probe AI Identity and Whether Models Disclose It](https://mlsanity.com/paper/2606.00168/) (2026)
- [Why Robust Natural Language Understanding is a Challenge](https://mlsanity.com/paper/2206.14575/) (2022)

## Abstract (arXiv)

Humans are increasingly interacting with machines through language, sometimes in contexts where the user may not know they are talking to a machine (like over the phone or a text chatbot). We aim to understand how system designers and researchers might allow their systems to confirm its non-human identity. We collect over 2,500 phrasings related to the intent of ``Are you a robot?". This is paired with over 2,500 adversarially selected utterances where only confirming the system is non-human would be insufficient or disfluent. We compare classifiers to recognize the intent and discuss the precision/recall and model complexity tradeoffs. Such classifiers could be integrated into dialog systems to avoid undesired deception. We then explore how both a generative research model (Blender) as well as two deployed systems (Amazon Alexa, Google Assistant) handle this intent, finding that systems often fail to confirm their non-human identity. Finally, we try to understand what a good response to the intent would be, and conduct a user study to compare the important aspects when responding to this intent.