# Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data

arXiv: https://arxiv.org/abs/2112.08878 · HTML: https://mlsanity.com/paper/2112.08878/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Large-Scale Domain Adaptation via Teacher-Student Learning](https://mlsanity.com/paper/1708.05466/) (2017)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Very Deep Multilingual Convolutional Neural Networks for LVCSR](https://mlsanity.com/paper/1509.08967/) (2015)
- [Recurrent Neural Network Training with Dark Knowledge Transfer](https://mlsanity.com/paper/1505.04630/) (2015)
- [Improving noise robustness of automatic speech recognition via parallel data and teacher-student learning](https://mlsanity.com/paper/1901.02348/) (2019)
- [Blending LSTMs into CNNs](https://mlsanity.com/paper/1511.06433/) (2015)
- [Transferring Knowledge from a RNN to a DNN](https://mlsanity.com/paper/1504.01483/) (2015)

## Built on by (most cited first, 1 of 1)

- [Low-latency Real-time Voice Conversion on CPU](https://mlsanity.com/paper/2311.00873/) (2023)

## Abstract (arXiv)

This paper describes a novel knowledge distillation framework that leverages acoustically qualified speech data included in an existing training data pool as privileged information. In our proposed framework, a student network is trained with multiple soft targets for each utterance that consist of main soft targets from original speakers' utterance and alternative targets from other speakers' utterances spoken under better acoustic conditions as a secondary view. These qualified utterances from other speakers, used to generate better soft targets, are collected from a qualified data pool by using strict constraints in terms of word/phone/state durations. Our proposed method is a form of target-side data augmentation that creates multiple copies of data with corresponding better soft targets obtained from a qualified data pool. We show in our experiments under acoustic model adaptation settings that the proposed method, exploiting better soft targets obtained from various speakers, can further improve recognition accuracy compared with conventional methods using only soft targets from original speakers.