TriNet: stabilizing self-supervised learning from complete or slow collapse on ASR
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) models confront challenges of abrupt informational collapse or slow dimensional collapse. We propose TriNet, which introduces a novel triple-branch architecture for preventing collapse and stabilizing the pre-training. TriNet learns the SSL latent embedding space and incorporates it to a higher level space for predicting pseudo target vectors generated by a frozen teacher. Our experimental results show that the proposed method notably stabilizes and accelerates pre-training and achieves a relative word error rate reduction (WERR) of 6.06% compared to the state-of-the-art (SOTA) Data2vec for a downstream benchmark ASR task. We will release our code at https://github.com/tencent-ailab/.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 12 Dec 2022.
Builds on
27 of 32 references.
References · 27
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
15 more
- 2020Whitening for Self-Supervised Representation Learningcited by 278
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
- 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
- 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
- 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
- 2019Self-Training for End-to-End Speech Recognitioncited by 128
- 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
- 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
- 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
- 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
- 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
- 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
- 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
- 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
- 2022Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learningcited by 10
Built on by
Citing papers per year: 2025: 1.
1 paper.