# From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

arXiv: https://arxiv.org/abs/2608.08510 · HTML: https://mlsanity.com/paper/2608.08510/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 62 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation](https://mlsanity.com/paper/2303.00628/) (2023)
- [MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding](https://mlsanity.com/paper/2106.01541/) (2021)
- [Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation](https://mlsanity.com/paper/2406.10082/) (2024)
- [Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion](https://mlsanity.com/paper/1603.09725/) (2016)
- [BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition](https://mlsanity.com/paper/2404.02098/) (2024)
- [An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue](https://mlsanity.com/paper/2501.16643/) (2025)
- [Speaker-Targeted Audio-Visual Models for Speech Recognition in Cocktail-Party Environments](https://mlsanity.com/paper/1906.05962/) (2019)
- [Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2009.09561/) (2020)
- [The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition](https://mlsanity.com/paper/2303.06326/) (2023)
- [Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction](https://mlsanity.com/paper/2310.19644/) (2023)
- [Multimodal Conversation Structure Understanding](https://mlsanity.com/paper/2505.17536/) (2025)
- [A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results](https://mlsanity.com/paper/2510.23276/) (2025)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder](https://mlsanity.com/paper/2601.18396/) (2026)
- [BUT System Description for CHiME-9 MCoRec Challenge](https://mlsanity.com/paper/2604.27436/) (2026)

## Abstract (arXiv)

Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.