# Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learning

arXiv: https://arxiv.org/abs/2305.18108 · HTML: https://mlsanity.com/paper/2305.18108/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2023.

## Built on by (most cited first, 28 of 28)

- [Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks](https://mlsanity.com/paper/2309.07937/) (2023)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation](https://mlsanity.com/paper/2312.02512/) (2023)
- [Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper](https://mlsanity.com/paper/2309.08535/) (2023)
- [SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks](https://mlsanity.com/paper/2408.13040/) (2024)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding](https://mlsanity.com/paper/2406.09345/) (2024)
- [LAST: Language Model Aware Speech Tokenization](https://mlsanity.com/paper/2409.03701/) (2024)
- [Estimating the Completeness of Discrete Speech Units](https://mlsanity.com/paper/2409.06109/) (2024)
- [Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations](https://mlsanity.com/paper/2407.03495/) (2024)
- [Exploring SSL Discrete Tokens for Multilingual ASR](https://mlsanity.com/paper/2409.08805/) (2024)
- [Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet](https://mlsanity.com/paper/2508.16576/) (2025)
- [Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens](https://mlsanity.com/paper/2309.08531/) (2023)
- [Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR](https://mlsanity.com/paper/2311.04534/) (2023)
- [Children's Speech Recognition through Discrete Token Enhancement](https://mlsanity.com/paper/2406.13431/) (2024)
- [On-device Streaming Discrete Speech Units](https://mlsanity.com/paper/2506.01845/) (2025)
- [Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing](https://mlsanity.com/paper/2309.15826/) (2023)
- [SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition](https://mlsanity.com/paper/2401.18045/) (2024)
- [A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models](https://mlsanity.com/paper/2411.08742/) (2024)
- [Discrete Speech Unit Extraction via Independent Component Analysis](https://mlsanity.com/paper/2501.06562/) (2025)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs](https://mlsanity.com/paper/2602.06180/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) of speech has shown impressive results in speech-related tasks, particularly in automatic speech recognition (ASR). While most methods employ the output of intermediate layers of the SSL model as real-valued features for downstream tasks, there is potential in exploring alternative approaches that use discretized token sequences. This approach offers benefits such as lower storage requirements and the ability to apply techniques from natural language processing. In this paper, we propose a new protocol that utilizes discretized token sequences in ASR tasks, which includes de-duplication and sub-word modeling to enhance the input sequence. It reduces computational cost by decreasing the length of the sequence. Our experiments on the LibriSpeech dataset demonstrate that our proposed protocol performs competitively with conventional ASR systems using continuous input features, while reducing computational and storage costs.