# Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization

arXiv: https://arxiv.org/abs/2512.18967 · HTML: https://mlsanity.com/paper/2512.18967/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models](https://mlsanity.com/paper/2310.02943/) (2023)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech](https://mlsanity.com/paper/2008.00702/) (2020)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition](https://mlsanity.com/paper/2307.08234/) (2023)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [End-to-end Speech-to-Punctuated-Text Recognition](https://mlsanity.com/paper/2207.03169/) (2022)
- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)
- [Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation](https://mlsanity.com/paper/2211.00508/) (2022)
- [Punctuation Prediction in Spontaneous Conversations: Can We Mitigate ASR Errors with Retrofitted Word Embeddings?](https://mlsanity.com/paper/2004.05985/) (2020)
- [End to End ASR System with Automatic Punctuation Insertion](https://mlsanity.com/paper/2012.02012/) (2020)

## Abstract (arXiv)

Conventional automatic speech recognition (ASR) models typically produce outputs as normalized texts lacking punctuation and capitalization, necessitating post-processing models to enhance readability. This approach, however, introduces additional complexity and latency due to the cascaded system design. In response to this challenge, there is a growing trend to develop end-to-end (E2E) ASR models capable of directly predicting punctuation and capitalization, though this area remains underexplored. In this paper, we propose an enhanced fully formatted E2E ASR model that leverages knowledge distillation (KD) through multi-codebook vector quantization (MVQ). Experimental results demonstrate that our model significantly outperforms previous works in word error rate (WER) both with and without punctuation and capitalization, and in punctuation error rate (PER). Evaluations on the LibriSpeech-PC test-clean and test-other subsets show that our model achieves state-of-the-art results.