# Modeling Speaker-Listener Interaction for Backchannel Prediction

arXiv: https://arxiv.org/abs/2304.04472 · HTML: https://mlsanity.com/paper/2304.04472/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 41 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [Exploring Semi-Supervised Learning for Predicting Listener Backchannels](https://mlsanity.com/paper/2101.01899/) (2021)
- [Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions](https://mlsanity.com/paper/1902.11060/) (2019)

## Built on by (most cited first, 3 of 3)

- [Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement](https://mlsanity.com/paper/2409.13726/) (2024)
- [Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction](https://mlsanity.com/paper/2607.22729/) (2026)
- [Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling](https://mlsanity.com/paper/2610.01488/) (2026)

## Abstract (arXiv)

We present our latest findings on backchannel modeling novelly motivated by the canonical use of the minimal responses Yeah and Uh-huh in English and their correspondent tokens in German, and the effect of encoding the speaker-listener interaction. Backchanneling theories emphasize the active and continuous role of the listener in the course of the conversation, their effects on the speaker's subsequent talk, and the consequent dynamic speaker-listener interaction. Therefore, we propose a neural-based acoustic backchannel classifier on minimal responses by processing acoustic features from the speaker speech, capturing and imitating listeners' backchanneling behavior, and encoding speaker-listener interaction. Our experimental results on the Switchboard and GECO datasets reveal that in almost all tested scenarios the speaker or listener behavior embeddings help the model make more accurate backchannel predictions. More importantly, a proper interaction encoding strategy, i.e., combining the speaker and listener embeddings, leads to the best performance on both datasets in terms of F1-score.