# UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models arXiv: https://arxiv.org/abs/2402.08898 ยท HTML: https://mlsanity.com/paper/2402.08898/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 37 references that are in the map. - First posted on arXiv 14 Feb 2024. ## Builds on (most cited first, 30 of 32) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018) - [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022) - [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021) - [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022) - [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021) - [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019) - [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020) - [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022) - [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021) - [SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training](https://mlsanity.com/paper/2210.03730/) (2022) - [An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04590/) (2021) - [Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment](https://mlsanity.com/paper/2010.14233/) (2020) - [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021) - [A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation](https://mlsanity.com/paper/2110.05249/) (2021) - [Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](https://mlsanity.com/paper/2205.01086/) (2022) - [CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition](https://mlsanity.com/paper/2010.14725/) (2020) - [Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR](https://mlsanity.com/paper/2305.00115/) (2023) - [Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models](https://mlsanity.com/paper/2201.10103/) (2022) - [Pushing the Limits of Non-Autoregressive Speech Recognition](https://mlsanity.com/paper/2104.03416/) (2021) - [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021) - [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022) - [An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2106.09885/) (2021) - [Understanding Shared Speech-Text Representations](https://mlsanity.com/paper/2304.14514/) (2023) ## Built on by (most cited first, 1 of 1) - [AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Models](https://mlsanity.com/paper/2609.23979/) (2026) ## Abstract (arXiv) Non-autoregressive automatic speech recognition (NASR) models have gained attention due to their parallelism and fast inference. The encoder-based NASR, e.g. connectionist temporal classification (CTC), can be initialized from the speech foundation models (SFM) but does not account for any dependencies among intermediate tokens. The encoder-decoder-based NASR, like CTC alignment-based single-step non-autoregressive transformer (CASS-NAT), can mitigate the dependency problem but is not able to efficiently integrate SFM. Inspired by the success of recent work of speech-text joint pre-training with a shared transformer encoder, we propose a new encoder-based NASR, UniEnc-CASSNAT, to combine the advantages of CTC and CASS-NAT. UniEnc-CASSNAT consists of only an encoder as the major module, which can be the SFM. The encoder plays the role of both the CASS-NAT encoder and decoder by two forward passes. The first pass of the encoder accepts the speech signal as input, while the concatenation of the speech signal and the token-level acoustic embedding is used as the input for the second pass. Examined on the Librispeech 100h, MyST, and Aishell1 datasets, the proposed UniEnc-CASSNAT achieves state-of-the-art NASR results and is better or comparable to CASS-NAT with only an encoder and hence, fewer model parameters. Our codes are publicly available.