# Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue

arXiv: https://arxiv.org/abs/2312.15316 · HTML: https://mlsanity.com/paper/2312.15316/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://mlsanity.com/paper/2304.09582/) (2023)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models](https://mlsanity.com/paper/1911.09762/) (2019)
- [What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation](https://mlsanity.com/paper/2203.13927/) (2022)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [Modulating Language Models with Emotions](https://mlsanity.com/paper/2108.07886/) (2021)
- [Incorporating End-to-End Speech Recognition Models for Sentiment Analysis](https://mlsanity.com/paper/1902.11245/) (2019)
- [User Response and Sentiment Prediction for Automatic Dialogue Evaluation](https://mlsanity.com/paper/2111.08808/) (2021)
- [Towards a Sentiment-Aware Conversational Agent](https://mlsanity.com/paper/2207.11774/) (2022)

## Built on by (most cited first, 25 of 25)

- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations](https://mlsanity.com/paper/2402.12786/) (2024)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2411.01834/) (2024)
- [Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech](https://mlsanity.com/paper/2410.01162/) (2024)
- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models](https://mlsanity.com/paper/2601.23255/) (2026)
- [Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation](https://mlsanity.com/paper/2408.11849/) (2024)
- [Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?](https://mlsanity.com/paper/2406.11065/) (2024)
- [Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights](https://mlsanity.com/paper/2406.17430/) (2024)
- [AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues](https://mlsanity.com/paper/2412.17292/) (2024)
- [Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations](https://mlsanity.com/paper/2503.12115/) (2025)
- [Towards Reliable Large Audio Language Model](https://mlsanity.com/paper/2505.19294/) (2025)
- [Resurfacing Paralinguistic Awareness in Large Audio Language Models](https://mlsanity.com/paper/2603.11947/) (2026)
- [Enhancing Speech Large Language Models through Reinforced Behavior Alignment](https://mlsanity.com/paper/2509.03526/) (2025)
- [Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2603.15981/) (2026)
- [VoiceLongMemEval: Do Assistants Remember How You Sounded?](https://mlsanity.com/paper/2609.00570/) (2026)
- [Dual Information Speech Language Models for Emotional Conversations](https://mlsanity.com/paper/2508.08095/) (2025)
- [Serial-Parallel Dual-Path Architecture for Speaking Style Recognition](https://mlsanity.com/paper/2510.11732/) (2025)
- [EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis](https://mlsanity.com/paper/2606.24941/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.24430/) (2026)
- [ER-EDF: A Psychology-Grounded Emotion Regulation Framework for Speech Empathetic Dialogue Generation in Large Audio-Language Models](https://mlsanity.com/paper/2609.15089/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralinguistic information, such as sentiment, emotion, and speaking style, which are essential for achieving natural, human-like spoken conversation, especially when such information is conveyed by acoustic cues. We therefore propose Paralinguistics-enhanced Generative Pretrained Transformer (ParalinGPT), an LLM that utilizes text and speech modalities to better model the linguistic content and paralinguistic attributes of spoken dialogue. The model takes the conversational context of text, speech embeddings, and paralinguistic attributes as input prompts within a serialized multitasking multimodal framework. Specifically, our framework serializes tasks in the order of current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation with autoregressive conditioning. We utilize the Switchboard-1 corpus, including its sentiment labels as the paralinguistic attribute, as our spoken dialogue dataset. Experimental results indicate the proposed serialized multitasking method outperforms typical sequence classification techniques on current and response sentiment classification. Furthermore, leveraging conversational context and speech embeddings significantly improves both response text generation and sentiment prediction. Our proposed framework achieves relative improvements of 6.7%, 12.0%, and 3.5% in current sentiment accuracy, response sentiment accuracy, and response text BLEU score, respectively.