mlsanityData to 7 Oct 2026

Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognition

arXiv 1811.04531 ↗First posted 12 Nov 2018cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We investigate the feasibility of sequence-level knowledge distillation of Sequence-to-Sequence (Seq2Seq) models for Large Vocabulary Continuous Speech Recognition (LVSCR). We first use a pre-trained larger teacher model to generate multiple hypotheses per utterance with beam search. With the same input, we then train the student model using these hypotheses generated from the teacher as pseudo labels in place of the original ground truth labels. We evaluate our proposed method using Wall Street Journal (WSJ) corpus. It achieved up to 9.8 × parameter reduction with accuracy loss of up to 7.0% word-error rate (WER) increase

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 0; 2023: 1; 2024: 2.

9 papers.

  1. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 13
  2. 2021Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillationcited by 12
  3. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  4. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 11
  5. 2021Exploiting Large-scale Teacher-Student Training for On-device Acoustic Modelscited by 5
  6. 2021Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenchescited by 2
  7. 2021Sequence-level self-learning with multiple hypothesescited by 0
  8. 2023Unraveling Key Factors of Knowledge Distillationcited by 0
  9. 2024An Efficient Self-Learning Framework For Interactive Spoken Dialog Systemscited by 0