# LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks

arXiv: https://arxiv.org/abs/2406.09153 · HTML: https://mlsanity.com/paper/2406.09153/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Learning by Aligning Videos in Time](https://mlsanity.com/paper/2103.17260/) (2021)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Differentiable Divergences Between Time Series](https://mlsanity.com/paper/2010.08354/) (2020)
- [Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering](https://mlsanity.com/paper/2305.11072/) (2023)
- [Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation](https://mlsanity.com/paper/2203.16104/) (2022)
- [DeepNAG: Deep Non-Adversarial Gesture Generation](https://mlsanity.com/paper/2011.09149/) (2020)
- [Soft Dynamic Time Warping for Multi-Pitch Estimation and Beyond](https://mlsanity.com/paper/2304.05032/) (2023)
- [SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations](https://mlsanity.com/paper/2403.06260/) (2024)
- [Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations](https://mlsanity.com/paper/2403.08738/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport](https://mlsanity.com/paper/2502.01588/) (2025)
- [Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations](https://mlsanity.com/paper/2601.21084/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL)-based speech models are extensively used for full-stack speech processing. However, it has been observed that improving SSL-based speech representations using unlabeled speech for content-related tasks is challenging and computationally expensive. Recent attempts have been made to address this issue with cost-effective self-supervised fine-tuning (SSFT) approaches. Continuing in this direction, a cost-effective SSFT method named "LASER: Learning by Aligning Self-supervised Representations" is presented. LASER is based on the soft-DTW alignment loss with temporal regularisation term. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR). A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only < 3 hours of fine-tuning on a single GPU.