# Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding

arXiv: https://arxiv.org/abs/2205.00693 · HTML: https://mlsanity.com/paper/2205.00693/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [ReSSL: Relational Self-Supervised Learning with Weak Augmentation](https://mlsanity.com/paper/2107.09282/) (2021)
- [ASR Error Correction and Domain Adaptation Using Machine Translation](https://mlsanity.com/paper/2003.07692/) (2020)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)
- [Error Correction in ASR using Sequence-to-Sequence Models](https://mlsanity.com/paper/2202.01157/) (2022)
- [Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning](https://mlsanity.com/paper/2111.14585/) (2021)
- [Adapting Pretrained Transformer to Lattices for Spoken Language Understanding](https://mlsanity.com/paper/2011.00780/) (2020)
- [N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses](https://mlsanity.com/paper/2106.06519/) (2021)
- [Learning Spoken Language Representations with Neural Lattice Language Modeling](https://mlsanity.com/paper/2007.02629/) (2020)

## Built on by (most cited first, 4 of 4)

- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Improve Text Classification Accuracy with Intent Information](https://mlsanity.com/paper/2212.07649/) (2022)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)

## Abstract (arXiv)

Spoken language understanding (SLU) is an essential task for machines to understand human speech for better interactions. However, errors from the automatic speech recognizer (ASR) usually hurt the understanding performance. In reality, ASR systems may not be easy to adjust for the target scenarios. Therefore, this paper focuses on learning utterance representations that are robust to ASR errors using a contrastive objective, and further strengthens the generalization ability by combining supervised contrastive learning and self-distillation in model fine-tuning. Experiments on three benchmark datasets demonstrate the effectiveness of our proposed approach.