# What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition

arXiv: https://arxiv.org/abs/2107.02294 · HTML: https://mlsanity.com/paper/2107.02294/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation](https://mlsanity.com/paper/2001.00891/) (2020)
- [BP-Transformer: Modelling Long-Range Context via Binary Partitioning](https://mlsanity.com/paper/1911.04070/) (2019)
- [Dialogue Act Classification with Context-Aware Self-Attention](https://mlsanity.com/paper/1904.02594/) (2019)
- [Guiding attention in Sequence-to-sequence models for Dialogue Act prediction](https://mlsanity.com/paper/2002.08801/) (2020)
- [Dialogue Act Sequence Labeling using Hierarchical encoder with CRF](https://mlsanity.com/paper/1709.04250/) (2017)
- [Lexico-acoustic Neural-based Models for Dialog Act Classification](https://mlsanity.com/paper/1803.00831/) (2018)
- [A Dual-Attention Hierarchical Recurrent Neural Network for Dialogue Act Classification](https://mlsanity.com/paper/1810.09154/) (2018)
- [A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks](https://mlsanity.com/paper/1805.06280/) (2018)
- [Neural-based Context Representation Learning for Dialog Act Classification](https://mlsanity.com/paper/1708.02561/) (2017)
- [End-to-end speech-to-dialog-act recognition](https://mlsanity.com/paper/2004.11419/) (2020)
- [Multi-Task Learning for Domain-General Spoken Disfluency Detection in Dialogue Systems](https://mlsanity.com/paper/1810.03352/) (2018)
- [Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition](https://mlsanity.com/paper/2003.06044/) (2020)

## Built on by (most cited first, 5 of 5)

- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)
- [A Deeper (Autoregressive) Approach to Non-Convergent Discourse Parsing](https://mlsanity.com/paper/2305.12510/) (2023)
- [DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs](https://mlsanity.com/paper/2605.31432/) (2026)
- [A Personalized Utterance Style (PUS) based Dialogue Strategy for Efficient Service Requirement Elicitation](https://mlsanity.com/paper/2301.04582/) (2023)
- [Understanding Clinical Cognitive Dialogues Using Large Language Models](https://mlsanity.com/paper/2609.34125/) (2026)

## Abstract (arXiv)

Dialog acts can be interpreted as the atomic units of a conversation, more fine-grained than utterances, characterized by a specific communicative function. The ability to structure a conversational transcript as a sequence of dialog acts -- dialog act recognition, including the segmentation -- is critical for understanding dialog. We apply two pre-trained transformer models, XLNet and Longformer, to this task in English and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora with dialog act segmentation error rates (DSER) of 8.4% and 14.2%. To understand the key factors affecting dialog act recognition, we perform a comparative analysis of models trained under different conditions. We find that the inclusion of a broader conversational context helps disambiguate many dialog act classes, especially those infrequent in the training data. The presence of punctuation in the transcripts has a massive effect on the models' performance, and a detailed analysis reveals specific segmentation patterns observed in its absence. Finally, we find that the label set specificity does not affect dialog act segmentation performance. These findings have significant practical implications for spoken language understanding applications that depend heavily on a good-quality segmentation being available.