# How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications

arXiv: https://arxiv.org/abs/2203.16822 · HTML: https://mlsanity.com/paper/2203.16822/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Learning Robust and Multilingual Speech Representations](https://mlsanity.com/paper/2001.11128/) (2020)
- [A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2201.08930/) (2022)
- [Don't speak too fast: The impact of data bias on self-supervised speech models](https://mlsanity.com/paper/2110.07957/) (2021)
- [Automatic Speech Recognition Benchmark for Air-Traffic Communications](https://mlsanity.com/paper/2006.10304/) (2020)
- [ASR4REAL: An extended benchmark for speech models](https://mlsanity.com/paper/2110.08583/) (2021)
- [Contextual Semi-Supervised Learning: An Approach To Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systems](https://mlsanity.com/paper/2104.03643/) (2021)
- [XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition](https://mlsanity.com/paper/2103.08207/) (2021)
- [Improving callsign recognition with air-surveillance data in air-traffic communication](https://mlsanity.com/paper/2108.12156/) (2021)
- [Endpoint Detection for Streaming End-to-End Multi-talker ASR](https://mlsanity.com/paper/2201.09979/) (2022)
- [A two-step approach to leverage contextual data: speech recognition in air-traffic communications](https://mlsanity.com/paper/2202.03725/) (2022)
- [Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model](https://mlsanity.com/paper/2104.02558/) (2021)
- [Lattice-Free MMI Adaptation Of Self-Supervised Pretrained Acoustic Models](https://mlsanity.com/paper/2012.14252/) (2020)

## Built on by (most cited first, 30 of 30)

- [ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications](https://mlsanity.com/paper/2211.04054/) (2022)
- [Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study](https://mlsanity.com/paper/2303.06740/) (2023)
- [Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition](https://mlsanity.com/paper/2208.01157/) (2022)
- [Improving Accented Speech Recognition with Multi-Domain Training](https://mlsanity.com/paper/2303.07924/) (2023)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [Implementing contextual biasing in GPU decoder for online ASR](https://mlsanity.com/paper/2306.15685/) (2023)
- [Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech](https://mlsanity.com/paper/2407.13035/) (2024)
- [A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers](https://mlsanity.com/paper/2304.07842/) (2023)
- [Some voices are too common: Building fair speech recognition systems using the Common Voice dataset](https://mlsanity.com/paper/2306.03773/) (2023)
- [Automatic Speech Recognition (ASR) for the Diagnosis of pronunciation of Speech Sound Disorders in Korean children](https://mlsanity.com/paper/2403.08187/) (2024)
- [Speech and Natural Language Processing Technologies for Pseudo-Pilot Simulator](https://mlsanity.com/paper/2212.07164/) (2022)
- [Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio](https://mlsanity.com/paper/2310.10922/) (2023)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptation](https://mlsanity.com/paper/2510.24570/) (2025)
- [Political corpus creation through automatic speech recognition on EU debates](https://mlsanity.com/paper/2304.08137/) (2023)
- [Lessons Learned in ATCO2: 5000 hours of Air Traffic Control Communications for Robust Automatic Speech Recognition and Understanding](https://mlsanity.com/paper/2305.01155/) (2023)
- [Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper](https://mlsanity.com/paper/2409.13499/) (2024)
- [In-domain SSL pre-training and streaming ASR](https://mlsanity.com/paper/2509.12101/) (2025)
- [Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter](https://mlsanity.com/paper/2309.09443/) (2023)
- [GMTR: Graph Matching Transformers](https://mlsanity.com/paper/2311.08141/) (2023)
- [DarkShot: Lighting Dark Images with Low-Compute and High-Quality](https://mlsanity.com/paper/2312.16805/) (2023)
- [TRLS: A Time Series Representation Learning Framework via Spectrogram for Medical Signal Processing](https://mlsanity.com/paper/2401.05431/) (2024)
- [TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks](https://mlsanity.com/paper/2401.05432/) (2024)
- [T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2403.13663/) (2024)
- [Strategic Arms with Side Communication Prevail Over Low-Regret MAB Algorithms](https://mlsanity.com/paper/2408.17101/) (2024)
- [Towards General Deepfake Detection with Dynamic Curriculum](https://mlsanity.com/paper/2410.11162/) (2024)
- [Attr-Int: A Simple and Effective Entity Alignment Framework for Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2410.13409/) (2024)
- [End-to-End Transformer-based Automatic Speech Recognition for Northern Kurdish: A Pioneering Approach](https://mlsanity.com/paper/2410.16330/) (2024)
- [Mask6D: Masked Pose Priors For 6D Object Pose Estimation](https://mlsanity.com/paper/2507.06486/) (2025)
- [Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition](https://mlsanity.com/paper/2606.21340/) (2026)

## Abstract (arXiv)

Recent work on self-supervised pre-training focus on leveraging large-scale unlabeled speech data to build robust end-to-end (E2E) acoustic models (AM) that can be later fine-tuned on downstream tasks e.g., automatic speech recognition (ASR). Yet, few works investigated the impact on performance when the data properties substantially differ between the pre-training and fine-tuning phases, termed domain shift. We target this scenario by analyzing the robustness of Wav2Vec 2.0 and XLS-R models on downstream ASR for a completely unseen domain, air traffic control (ATC) communications. We benchmark these two models on several open-source and challenging ATC databases with signal-to-noise ratio between 5 and 20 dB. Relative word error rate (WER) reductions between 20% to 40% are obtained in comparison to hybrid-based ASR baselines by only fine-tuning E2E acoustic models with a smaller fraction of labeled data. We analyze WERs on the low-resource scenario and gender bias carried by one ATC dataset.