# CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning

arXiv: https://arxiv.org/abs/2210.04062 · HTML: https://mlsanity.com/paper/2210.04062/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2022.

## Built on by (most cited first, 7 of 7)

- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [PolyVoice: Language Models for Speech to Speech Translation](https://mlsanity.com/paper/2306.02982/) (2023)
- [DUB: Discrete Unit Back-translation for Speech Translation](https://mlsanity.com/paper/2305.11411/) (2023)
- [Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2309.13860/) (2023)
- [Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation](https://mlsanity.com/paper/2306.08920/) (2023)
- [Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs](https://mlsanity.com/paper/2511.16639/) (2025)

## Abstract (arXiv)

Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task. Our code and models are released at https://github.com/mct10/CoBERT.