# Uncertainty Modeling in Multimodal Speech Analysis Across the Psychosis Spectrum

arXiv: https://arxiv.org/abs/2502.18285 · HTML: https://mlsanity.com/paper/2502.18285/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 77 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Topic Modeling Based Multi-modal Depression Detection](https://mlsanity.com/paper/1803.10384/) (2018)
- [Artificial Intelligence, speech and language processing approaches to monitoring Alzheimer's Disease: a systematic review](https://mlsanity.com/paper/2010.06047/) (2020)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [Leveraging Large Language Models for Exploiting ASR Uncertainty](https://mlsanity.com/paper/2309.04842/) (2023)
- [Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition](https://mlsanity.com/paper/2407.01143/) (2024)
- [Uncertainty Quantification in Machine Learning for Joint Speaker Diarization and Identification](https://mlsanity.com/paper/2312.16763/) (2023)

## Built on by (most cited first, 1 of 1)

- [Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysis](https://mlsanity.com/paper/2511.15512/) (2025)

## Abstract (arXiv)

Capturing subtle speech disruptions across the psychosis spectrum is challenging because of the inherent variability in speech patterns. This variability reflects individual differences and the fluctuating nature of symptoms in both clinical and non-clinical populations. Accounting for uncertainty in speech data is essential for predicting symptom severity and improving diagnostic precision. Speech disruptions characteristic of psychosis appear across the spectrum, including in non-clinical individuals. We develop an uncertainty-aware model integrating acoustic and linguistic features to predict symptom severity and psychosis-related traits. Quantifying uncertainty in specific modalities allows the model to address speech variability, improving prediction accuracy. We analyzed speech data from 114 participants, including 32 individuals with early psychosis and 82 with low or high schizotypy, collected through structured interviews, semi-structured autobiographical tasks, and narrative-driven interactions in German. The model improved prediction accuracy, reducing RMSE and achieving an F1-score of 83% with ECE = 4.5e-2, showing robust performance across different interaction contexts. Uncertainty estimation improved model interpretability by identifying reliability differences in speech markers such as pitch variability, fluency disruptions, and spectral instability. The model dynamically adjusted to task structures, weighting acoustic features more in structured settings and linguistic features in unstructured contexts. This approach strengthens early detection, personalized assessment, and clinical decision-making in psychosis-spectrum research.