mlsanityData to 14 Oct 2026

Enhance Language Identification using Dual-mode Model with Knowledge Distillation

arXiv 2203.03218 ↗First posted 7 Mar 2022eess.AS cs.CL cs.SDcited by 1

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

In this paper, we propose to employ a dual-mode framework on the x-vector self-attention (XSA-LID) model with knowledge distillation (KD) to enhance its language identification (LID) performance for both long and short utterances. The dual-mode XSA-LID model is trained by jointly optimizing both the full and short modes with their respective inputs being the full-length speech and its short clip extracted by a specific Boolean mask, and KD is applied to further boost the performance on short utterances. In addition, we investigate the impact of clip-wise linguistic variability and lexical integrity for LID by analyzing the variation of LID performance in terms of the lengths and positions of the mimicked speech clips. We evaluated our approach on the MLS14 data from the NIST 2017 LRE. With the 3~s random-location Boolean mask, our proposed method achieved 19.23%, 21.52% and 8.37% relative improvement in average cost compared with the XSA-LID model on 3s, 10s, and 30s speech, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 7 of its 25 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

7 of 25 references.

References · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  5. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  6. 2019Utterance-level end-to-end language identification using attention-based CNN-BLSTMcited by 4
  7. 2018Tuplemax Loss for Language Identificationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Cross-Corpora Spoken Language Identification with Domain Diversification and Generalizationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach
  2. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
  3. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  4. 2021Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data
  5. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  6. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  7. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  8. 2023Adaptive Knowledge Distillation between Text and Speech Pre-trained Models