# Advances in Online Audio-Visual Meeting Transcription

arXiv: https://arxiv.org/abs/1912.04979 · HTML: https://mlsanity.com/paper/1912.04979/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 23 of 23)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021)
- [NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription](https://mlsanity.com/paper/2401.08887/) (2024)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio](https://mlsanity.com/paper/2107.02852/) (2021)
- [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021)
- [Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization](https://mlsanity.com/paper/2210.07764/) (2022)
- [Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordings](https://mlsanity.com/paper/2101.01853/) (2021)
- [Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.11232/) (2022)
- [Continuous Speech Separation with Ad Hoc Microphone Arrays](https://mlsanity.com/paper/2103.02378/) (2021)
- [Speech separation with large-scale self-supervised learning](https://mlsanity.com/paper/2211.05172/) (2022)
- [Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation](https://mlsanity.com/paper/2204.03232/) (2022)
- [Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts](https://mlsanity.com/paper/2211.06493/) (2022)
- [Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications](https://mlsanity.com/paper/2403.06570/) (2024)
- [Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization](https://mlsanity.com/paper/2408.12102/) (2024)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation](https://mlsanity.com/paper/2505.13094/) (2025)
- [Diarisation using location tracking with agglomerative clustering](https://mlsanity.com/paper/2109.10598/) (2021)
- [Joint speaker diarisation and tracking in switching state-space model](https://mlsanity.com/paper/2109.11140/) (2021)

## Abstract (arXiv)

This paper describes a system that generates speaker-annotated transcripts of meetings by using a microphone array and a 360-degree camera. The hallmark of the system is its ability to handle overlapped speech, which has been an unsolved problem in realistic settings for over a decade. We show that this problem can be addressed by using a continuous speech separation approach. In addition, we describe an online audio-visual speaker diarization method that leverages face tracking and identification, sound source localization, speaker identification, and, if available, prior speaker information for robustness to various real world challenges. All components are integrated in a meeting transcription framework called SRD, which stands for "separate, recognize, and diarize". Experimental results using recordings of natural meetings involving up to 11 attendees are reported. The continuous speech separation improves a word error rate (WER) by 16.1% compared with a highly tuned beamformer. When a complete list of meeting attendees is available, the discrepancy between WER and speaker-attributed WER is only 1.0%, indicating accurate word-to-speaker association. This increases marginally to 1.6% when 50% of the attendees are unknown to the system.