mlsanityData to 14 Oct 2026

Temporal Knowledge Distillation for On-device Audio Classification

arXiv 2110.14131 ↗First posted 27 Oct 2021cs.SD cs.LG eess.AScited by 15

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Improving the performance of on-device audio classification models remains a challenge given the computational limits of the mobile environment. Many studies leverage knowledge distillation to boost predictive performance by transferring the knowledge from large models to on-device models. However, most lack a mechanism to distill the essence of the temporal information, which is crucial to audio classification tasks, or similar architecture is often required. In this paper, we propose a new knowledge distillation method designed to incorporate the temporal knowledge embedded in attention weights of large transformer-based models into on-device models. Our distillation method is applicable to various types of architectures, including the non-attention-based architectures such as CNNs or RNNs, while retaining the original network architecture during inference. Through extensive experiments on both an audio event detection dataset and a noisy keyword spotting dataset, we show that our proposed method improves the predictive performance across diverse on-device architectures.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 19 of its 22 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

19 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition14 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  6. 2020Quantifying Attention Flow in Transformerscited by 775
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  9. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  10. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 60
  11. 2019Temporal Convolution for Real-time Keyword Spotting on Mobile Devicescited by 49
  12. 2020Streaming keyword spotting on mobile devicescited by 47
6 more
  1. 2018A neural attention model for speech command recognitioncited by 45
  2. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 41
  3. 2017Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spottingcited by 32
  4. 2016Knowledge Distillation for Small-footprint Highway Networkscited by 13
  5. 2018Sequence-Level Knowledge Distillation for Model Compression of Attention-based Sequence-to-Sequence Speech Recognitioncited by 9
  6. 2020Intra-Utterance Similarity Preserving Knowledge Distillation for Audio Taggingcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 5; 2024: 3; 2025: 2.

15 papers.

Sort:

By area:

  1. 2022Opening the Black Box of wav2vec Feature Encodercited by 18
  2. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  3. 2022Simple Pooling Front-ends For Efficient Audio Classificationcited by 9
  4. 2022Distilling a Pretrained Language Model to a Multilingual ASR Modelcited by 7
  5. 2024Generalization in birdsong classification: impact of transfer learning methods and dataset characteristicscited by 5
  6. 2022Continual Learning For On-Device Environmental Sound Classificationcited by 4
  7. 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understandingcited by 3
  8. 2024A Predictive Approach To Enhance Time-Series Forecastingcited by 3
  9. 2023Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classificationcited by 1
  10. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  11. 2023Content Adaptive Front End For Audio Classificationcited by 0
  12. 2023Knowledge Distillation for Efficient Audio-Visual Video Captioningcited by 0
3 more
  1. 2023Diverse Audio Embeddings -- Bringing Features Back Outperforms CLAP!cited by 0
  2. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  3. 2025Structural and Statistical Audio Texture Knowledge Distillation for Acoustic Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Knowledge Distillation for Real-Time Classification of Early Media in Voice Communications
  2. 2024Dual Knowledge Distillation for Efficient Sound Event Detection
  3. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation
  4. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
  5. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detection
  6. 2025Exploring Performance-Complexity Trade-Offs in Sound Event Detection Models
  7. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learning
  8. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models