# Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition

arXiv: https://arxiv.org/abs/2609.20081 · HTML: https://mlsanity.com/paper/2609.20081/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution](https://mlsanity.com/paper/2403.03121/) (2024)
- [EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark](https://mlsanity.com/paper/2406.07162/) (2024)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Label Supervised LLaMA Finetuning](https://mlsanity.com/paper/2310.01208/) (2023)
- [Speech Emotion Recognition using Supervised Deep Recurrent System for Mental Health Monitoring](https://mlsanity.com/paper/2208.12812/) (2022)
- [Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques](https://mlsanity.com/paper/2406.08353/) (2024)
- [Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models](https://mlsanity.com/paper/2407.06908/) (2024)
- [LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2501.11468/) (2025)
- [End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations](https://mlsanity.com/paper/2310.02281/) (2023)
- [Emotion Neural Transducer for Fine-Grained Speech Emotion Recognition](https://mlsanity.com/paper/2403.19224/) (2024)
- [EmoQ: Speech Emotion Recognition via Speech-Aware Q-Former and Large Language Model](https://mlsanity.com/paper/2509.15775/) (2025)
- [Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2606.05713/) (2026)
- [Effective Context Modeling Framework for Emotion Recognition in Conversations](https://mlsanity.com/paper/2412.16444/) (2024)
- [Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models](https://mlsanity.com/paper/2510.02569/) (2025)
- [A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification](https://mlsanity.com/paper/2512.07571/) (2025)
- [Enhancing Speech Emotion Recognition with Multi-Task Learning and Dynamic Feature Fusion](https://mlsanity.com/paper/2508.17878/) (2025)

## Abstract (arXiv)

SpeechLLMs have shown strong potential for emotion recognition, yet they read the predicted emotion off a generative decoder not suited for classification: it can emit labels outside the target set and favors frequent classes. We propose a discriminative adaptation that reads the final prompt token's hidden state through a classification head, producing a label in one forward pass without modifying the backbone. Because this readout starts from the hidden state the model would otherwise decode, it gives a controlled comparison of generative and discriminative inference in an otherwise identical speechLLM. We keep the head a single linear layer, trading little accuracy for interpretability: each emotion becomes one direction in the LLM output token space, revealing associated tokens. On IEMOCAP, across two speechLLM architectures, it improves Macro F1 and removes hallucinations, with largest gains on realistic ASR transcripts. Our analysis reveals that these emotion directions encode indirect associations mirroring biases in web-scale text.