# CTCBERT: Advancing Hidden-unit BERT with CTC Objectives

arXiv: https://arxiv.org/abs/2210.08603 · HTML: https://mlsanity.com/paper/2210.08603/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2022.

## Built on by (most cited first, 5 of 5)

- [Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2309.13860/) (2023)
- [Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation](https://mlsanity.com/paper/2306.08920/) (2023)
- [VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining](https://mlsanity.com/paper/2505.21527/) (2025)
- [UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models](https://mlsanity.com/paper/2402.08898/) (2024)
- [Improving Self-supervised Pre-training using Accent-Specific Codebooks](https://mlsanity.com/paper/2407.03734/) (2024)

## Abstract (arXiv)

In this work, we present a simple but effective method, CTCBERT, for advancing hidden-unit BERT (HuBERT). HuBERT applies a frame-level cross-entropy (CE) loss, which is similar to most acoustic model training. However, CTCBERT performs the model training with the Connectionist Temporal Classification (CTC) objective after removing duplicated IDs in each masked region. The idea stems from the observation that there can be significant errors in alignments when using clustered or aligned IDs. CTC learns alignments implicitly, indicating that learning with CTC can be more flexible when misalignment exists. We examine CTCBERT on IDs from HuBERT Iter1, HuBERT Iter2, and PBERT. The CTC training brings consistent improvements compared to the CE training. Furthermore, when loading blank-related parameters during finetuning, slight improvements are observed. Evaluated on the Librispeech 960-100h setting, the relative WER improvements of CTCBERT are 2%-11% over HuBERT and PERT on test-other data.