# Speaker-Aware Simulation Improves Conversational Speech Recognition

arXiv: https://arxiv.org/abs/2602.04776 · HTML: https://mlsanity.com/paper/2602.04776/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 16 of 16)

- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.11232/) (2022)
- [Multi-Speaker and Wide-Band Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2211.06750/) (2022)
- [LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization](https://mlsanity.com/paper/2409.00819/) (2024)
- [From Independence to Interaction: Speaker-Aware Simulation of Multi-Speaker Conversational Timing](https://mlsanity.com/paper/2509.15808/) (2025)
- [Toward Conversational Hungarian Speech Recognition: Introducing the BEA-Large and BEA-Dialogue Datasets](https://mlsanity.com/paper/2511.13529/) (2025)
- [LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization](https://mlsanity.com/paper/2510.23320/) (2025)

## Built on by (most cited first, 2 of 2)

- [Efficient ASR Training with Conversations that Never Happened](https://mlsanity.com/paper/2606.03957/) (2026)
- [On the Role of Conversational Timing in Synthetic Training Data for ASR](https://mlsanity.com/paper/2607.08371/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) for conversational speech remains challenging due to the limited availability of large-scale, well-annotated multi-speaker dialogue data and the complex temporal dynamics of natural interactions. Speaker-aware simulated conversations (SASC) offer an effective data augmentation strategy by transforming single-speaker recordings into realistic multi-speaker dialogues. However, prior work has primarily focused on English data, leaving questions about the applicability to lower-resource languages. In this paper, we adapt and implement the SASC framework for Hungarian conversational ASR. We further propose C-SASC, an extended variant that incorporates pause modeling conditioned on utterance duration, enabling a more faithful representation of local temporal dependencies observed in human conversation while retaining the simplicity and efficiency of the original approach. We generate synthetic Hungarian dialogues from the BEA-Large corpus and combine them with real conversational data for ASR training. Both SASC and C-SASC are evaluated extensively under a wide range of simulation configurations, using conversational statistics derived from CallHome, BEA-Dialogue, and GRASS corpora. Experimental results show that speaker-aware conversational simulation consistently improves recognition performance over naive concatenation-based augmentation. While the additional duration conditioning in C-SASC yields modest but systematic gains--most notably in character-level error rates--its effectiveness depends on the match between source conversational statistics and the target domain. Overall, our findings confirm the robustness of speaker-aware conversational simulation for Hungarian ASR and highlight the benefits and limitations of increasingly detailed temporal modeling in synthetic dialogue generation.