# Improving End-to-End SLU performance with Prosodic Attention and Distillation

arXiv: https://arxiv.org/abs/2305.08067 · HTML: https://mlsanity.com/paper/2305.08067/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [Self-attention encoding and pooling for speaker recognition](https://mlsanity.com/paper/2008.01077/) (2020)
- [STOP: A dataset for Spoken Task Oriented Semantic Parsing](https://mlsanity.com/paper/2207.10643/) (2022)
- [Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification](https://mlsanity.com/paper/2108.02598/) (2021)
- [It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system](https://mlsanity.com/paper/2105.00260/) (2021)
- [A neural prosody encoder for end-ro-end dialogue act classification](https://mlsanity.com/paper/2205.05590/) (2022)
- [Skit-S2I: An Indian Accented Speech to Intent dataset](https://mlsanity.com/paper/2212.13015/) (2022)
- [Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2204.05188/) (2022)
- [Shennong: a Python toolbox for audio speech features extraction](https://mlsanity.com/paper/2112.05555/) (2021)

## Built on by (most cited first, 5 of 5)

- [The Role of Prosody in Spoken Question Answering](https://mlsanity.com/paper/2502.05389/) (2025)
- [QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding](https://mlsanity.com/paper/2505.14723/) (2025)
- [Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody](https://mlsanity.com/paper/2506.02057/) (2025)
- [Pitch Accent Detection improves Pretrained Automatic Speech Recognition](https://mlsanity.com/paper/2508.04814/) (2025)
- [Emphasis Sensitivity in Speech Representations](https://mlsanity.com/paper/2508.11566/) (2025)

## Abstract (arXiv)

Most End-to-End SLU methods depend on the pretrained ASR or language model features for intent prediction. However, other essential information in speech, such as prosody, is often ignored. Recent research has shown improved results in classifying dialogue acts by incorporating prosodic information. The margins of improvement in these methods are minimal as the neural models ignore prosodic features. In this work, we propose prosody-attention, which uses the prosodic features differently to generate attention maps across time frames of the utterance. Then we propose prosody-distillation to explicitly learn the prosodic information in the acoustic encoder rather than concatenating the implicit prosodic features. Both the proposed methods improve the baseline results, and the prosody-distillation method gives an intent classification accuracy improvement of 8% and 2% on SLURP and STOP datasets over the prosody baseline.