# ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations

arXiv: https://arxiv.org/abs/2507.13468 · HTML: https://mlsanity.com/paper/2507.13468/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 40 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 5 of 5)

- [User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants](https://mlsanity.com/paper/2309.13879/) (2023)
- [Modeling Human Response to Robot Errors for Timely Error Detection](https://mlsanity.com/paper/2208.00565/) (2022)
- [ERR@HRI 2024 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Interactions](https://mlsanity.com/paper/2407.06094/) (2024)
- ["Oh, Sorry, I Think I Interrupted You'': Designing Repair Strategies for Robotic Longitudinal Well-being Coaching](https://mlsanity.com/paper/2401.03794/) (2024)
- [Voice Assistants for Health Self-Management: Designing for and with Older Adults](https://mlsanity.com/paper/2409.15488/) (2024)

## Built on by (most cited first, 1 of 1)

- [Towards a Foundational Ontology for Identifying and Resolving Contradictions in Dialogue-based Human-Robot Interactions](https://mlsanity.com/paper/2609.02364/) (2026)

## Abstract (arXiv)

The integration of large language models (LLMs) into conversational robots has made human-robot conversations more dynamic. Yet, LLM-powered conversational robots remain prone to errors, e.g., misunderstanding user intent, prematurely interrupting users, or failing to respond altogether. Detecting and addressing these failures is critical for preventing conversational breakdowns, avoiding task disruptions, and sustaining user trust. To tackle this problem, the ERR@HRI 2.0 Challenge provides a multimodal dataset of LLM-powered conversational robot failures during human-robot conversations and encourages researchers to benchmark machine learning models designed to detect robot failures. The dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features. Each interaction is annotated with the presence or absence of robot errors from the system perspective, and perceived user intention to correct for a mismatch between robot behavior and user expectation. Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data. Submissions will be evaluated using various performance metrics, including detection accuracy and false positive rate. This challenge represents another key step toward improving failure detection in human-robot interaction through social signal analysis.