# A neural prosody encoder for end-ro-end dialogue act classification

arXiv: https://arxiv.org/abs/2205.05590 · HTML: https://mlsanity.com/paper/2205.05590/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Lexico-acoustic Neural-based Models for Dialog Act Classification](https://mlsanity.com/paper/1803.00831/) (2018)
- [It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system](https://mlsanity.com/paper/2105.00260/) (2021)
- [Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection](https://mlsanity.com/paper/1604.00077/) (2016)
- [End-to-end speech-to-dialog-act recognition](https://mlsanity.com/paper/2004.11419/) (2020)
- [Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation](https://mlsanity.com/paper/1810.07455/) (2018)

## Built on by (most cited first, 5 of 5)

- [Improving End-to-End SLU performance with Prosodic Attention and Distillation](https://mlsanity.com/paper/2305.08067/) (2023)
- [QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding](https://mlsanity.com/paper/2505.14723/) (2025)
- [Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody](https://mlsanity.com/paper/2506.02057/) (2025)
- [Pitch Accent Detection improves Pretrained Automatic Speech Recognition](https://mlsanity.com/paper/2508.04814/) (2025)
- [End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders](https://mlsanity.com/paper/2305.02937/) (2023)

## Abstract (arXiv)

Dialogue act classification (DAC) is a critical task for spoken language understanding in dialogue systems. Prosodic features such as energy and pitch have been shown to be useful for DAC. Despite their importance, little research has explored neural approaches to integrate prosodic features into end-to-end (E2E) DAC models which infer dialogue acts directly from audio signals. In this work, we propose an E2E neural architecture that takes into account the need for characterizing prosodic phenomena co-occurring at different levels inside an utterance. A novel part of this architecture is a learnable gating mechanism that assesses the importance of prosodic features and selectively retains core information necessary for E2E DAC. Our proposed model improves DAC accuracy by 1.07% absolute across three publicly available benchmark datasets.