# Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

arXiv: https://arxiv.org/abs/2607.23845 · HTML: https://mlsanity.com/paper/2607.23845/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 41 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 5 of 5)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance](https://mlsanity.com/paper/2504.08716/) (2025)
- ["Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue](https://mlsanity.com/paper/2510.24628/) (2025)

## Abstract (arXiv)

Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it. In the case of conversational agents, it is essential to accurately identify these repair initiation strategies to address communication breakdowns efficiently. While conversational analysis studies have shown that OIR initiation is accompanied by both verbal and non-verbal signals such as gaze shifts, facial expressions, body postures, and hand gestures, existing computational approaches rely mainly on text and audio. This paper introduces a novel multimodal model for OIR detection and classification, incorporating a set of visual features drawn from conversation analysis. We evaluate our approach on two corpora with distinct languages and interaction settings. Results demonstrate that visual information consistently improves performance over text and audio baselines, and provide insights into cross-modal feature contributions across two corpora.