# React to This! How Humans Challenge Interactive Agents using Nonverbal Behaviors

arXiv: https://arxiv.org/abs/2409.11602 · HTML: https://mlsanity.com/paper/2409.11602/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2024.

## Built on by (most cited first, 3 of 3)

- [EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation](https://mlsanity.com/paper/2512.22808/) (2025)
- [React to This (RTT): A Nonverbal Turing Test for Embodied AI](https://mlsanity.com/paper/2507.10812/) (2025)
- [SalsaAgent: A multimodal embodied language model for interactive dance generation](https://mlsanity.com/paper/2605.29219/) (2026)

## Abstract (arXiv)

How do people use their faces and bodies to test the interactive abilities of a robot? Making lively, believable agents is often seen as a goal for robots and virtual agents but believability can easily break down. In this Wizard-of-Oz (WoZ) study, we observed 1169 nonverbal interactions between 20 participants and 6 types of agents. We collected the nonverbal behaviors participants used to challenge the characters physically, emotionally, and socially. The participants interacted freely with humanoid and non-humanoid forms: a robot, a human, a penguin, a pufferfish, a banana, and a toilet. We present a human behavior codebook of 188 unique nonverbal behaviors used by humans to test the virtual characters. The insights and design strategies drawn from video observations aim to help build more interaction-aware and believable robots, especially when humans push them to their limits.