# A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results

arXiv: https://arxiv.org/abs/2510.23276 · HTML: https://mlsanity.com/paper/2510.23276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Large Language Models are Strong Audio-Visual Speech Recognition Learners](https://mlsanity.com/paper/2409.12319/) (2024)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation](https://mlsanity.com/paper/2303.00628/) (2023)
- [DiPCo -- Dinner Party Corpus](https://mlsanity.com/paper/1909.13447/) (2019)
- [Super-Human Performance in Online Low-latency Recognition of Conversational Speech](https://mlsanity.com/paper/2010.03449/) (2020)
- [Cocktail-Party Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.02178/) (2025)

## Built on by (most cited first, 4 of 4)

- [Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings](https://mlsanity.com/paper/2606.17542/) (2026)
- [From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios](https://mlsanity.com/paper/2608.08510/) (2026)
- [AVSRBench: A Multi-Condition AVSR Benchmark](https://mlsanity.com/paper/2609.10366/) (2026)
- [Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2609.10394/) (2026)

## Abstract (arXiv)

We introduce the task of Multi-Modal Context-Aware Recognition (MCoRec) in the ninth CHiME Challenge, which addresses the cocktail-party problem of overlapping conversations in a single-room setting using audio, visual, and contextual cues. MCoRec captures natural multi-party conversations where the recordings focus on unscripted, casual group chats, leading to extreme speech overlap of up to 100% and highly fragmented conversational turns. The task requires systems to answer the question "Who speaks when, what, and with whom?" by jointly transcribing each speaker's speech and clustering them into their respective conversations from audio-visual recordings. Audio-only baselines exceed 100% word error rate, whereas incorporating visual cues yields substantial 50% improvements, highlighting the importance of multi-modality. In this manuscript, we present the motivation behind the task, outline the data collection process, and report the baseline systems developed for the MCoRec.