# Learning Physiology-Informed Vocal Spectrotemporal Representations for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2602.13259 · HTML: https://mlsanity.com/paper/2602.13259/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Quaternion Convolutional Neural Networks](https://mlsanity.com/paper/1903.00658/) (2019)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Humanoid Robots and Humanoid AI: Review, Perspectives and Directions](https://mlsanity.com/paper/2405.15775/) (2024)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)

## Built on by (most cited first, 1 of 1)

- [DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech](https://mlsanity.com/paper/2606.00066/) (2026)

## Abstract (arXiv)

Speech emotion recognition (SER) is essential for humanoid robot tasks such as social robotic interactions and robotic psychological diagnosis, where interpretable and efficient models are critical for safety and performance. Existing deep models trained on large datasets remain largely uninterpretable, often insufficiently modeling underlying emotional acoustic signals and failing to capture and analyze the core physiology of emotional vocal behaviors. Physiological research on human voices shows that the dynamics of vocal amplitude and phase correlate with emotions through the vocal tract filter and the glottal source. However, most existing deep models solely involve amplitude but fail to couple the physiological features of and between amplitude and phase. Here, we propose PhysioSER, a physiology-informed vocal spectrotemporal representation learning method, to address these issues with a compact, plug-and-play design. PhysioSER constructs amplitude and phase views informed by voice anatomy and physiology (VAP) to complement SSL models for SER. This VAP-informed framework incorporates two parallel workflows: a vocal feature representation branch to decompose vocal signals based on VAP, embed them into a quaternion field, and use Hamilton-structured quaternion convolutions for modeling their dynamic interactions; and a latent representation branch based on a frozen SSL backbone. Then, utterance-level features from both workflows are aligned by a Contrastive Projection and Alignment framework, followed by a shallow attention fusion head for SER classification. PhysioSER is shown to be interpretable and efficient for SER through extensive evaluations across 14 datasets, 10 languages, and 6 backbones, and its practical efficacy is validated by real-time deployment on a humanoid robotic platform.