Contrastive Siamese Network for Semi-supervised Speech Recognition
Not on the map. Why?
Abstract (arXiv)
This paper introduces contrastive siamese (c-siam) network, an architecture for leveraging unlabeled acoustic data in speech recognition. c-siam is the first network that extracts high-level linguistic information from speech by matching outputs of two identical transformer encoders. It contains augmented and target branches which are trained by: (1) masking inputs and matching outputs with a contrastive loss, (2) incorporating a stop gradient operation on the target branch, (3) using an extra learnable transformation on the augmented branch, (4) introducing new temporal augment functions to prevent the shortcut learning problem. We use the Libri-light 60k unsupervised data and the LibriSpeech 100hrs/960hrs supervised data to compare c-siam and other best-performing systems. Our experiments show that c-siam provides 20% relative word error rate improvement over wav2vec baselines. A c-siam network with 450M parameters achieves competitive results compared to the state-of-the-art networks with 600M parameters.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 27 May 2022.
Builds on
22 of 27 references.
References · 22
- 2014Adam: A Method for Stochastic Optimizationcited by 37,093
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
- 2020Exploring Simple Siamese Representation Learningcited by 2,860
- 2020Shortcut Learning in Deep Neural Networkscited by 1,944
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,844
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,359
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,023
10 more
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 342
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 337
- 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
- 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
- 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 187
- 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
- 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
- 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
- 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
- 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.