mlsanityData to 7 Oct 2026

Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study

arXiv 2512.20948 ↗First posted 24 Dec 2025cs.CL cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Neuropsychiatric disorders, such as Alzheimer's disease (AD), depression, and autism spectrum disorder (ASD), are characterized by linguistic and acoustic abnormalities, offering potential biomarkers for early detection. Despite the promise of multi-modal approaches, challenges like multi-lingual generalization and the absence of a unified evaluation framework persist. To address these gaps, we propose FEND (Foundation model-based Evaluation of Neuropsychiatric Disorders), a comprehensive multi-modal framework integrating speech and text modalities for detecting AD, depression, and ASD across the lifespan. Leveraging 13 multi-lingual datasets spanning English, Chinese, Greek, French, and Dutch, we systematically evaluate multi-modal fusion performance. Our results show that multi-modal fusion excels in AD and depression detection but underperforms in ASD due to dataset heterogeneity. We also identify modality imbalance as a prevalent issue, where multi-modal fusion fails to surpass the best mono-modal models. Cross-corpus experiments reveal robust performance in task- and language-consistent scenarios but noticeable degradation in multi-lingual and task-heterogeneous settings. By providing extensive benchmarks and a detailed analysis of performance-influencing factors, FEND advances the field of automated, lifespan-inclusive, and multi-lingual neuropsychiatric disorder assessment. We encourage researchers to adopt the FEND framework for fair comparisons and reproducible research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 106 references that are in the map. First posted on arXiv 24 Dec 2025.

Builds on

59 of 106 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
38 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  4. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  7. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  8. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  9. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  11. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  12. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  13. 2018Learning Factorized Multimodal Representationscited by 171
  14. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  15. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  16. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  17. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  18. 2020Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challengecited by 104
  19. 2021Detecting cognitive decline using speech only: The ADReSSo Challengecited by 81
  20. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  21. 2020MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysiscited by 38
  22. 2022Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Modelcited by 37
  23. 2020Artificial Intelligence, speech and language processing approaches to monitoring Alzheimer's Disease: a systematic reviewcited by 29
  24. 2021Alzheimer's Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputscited by 27
  25. 2021Comparing Acoustic-based Approaches for Alzheimer's Disease Detectioncited by 21
  26. 2019Multi-level Attention network using text, audio and video for Depression Predictioncited by 19
  27. 2019MFCC-based Recurrent Neural Network for Automatic Clinical Depression Recognition and Assessment from Speechcited by 18
  28. 2023Multilingual Alzheimer's Dementia Recognition through Spontaneous Speech: a Signal Processing Grand Challengecited by 17
  29. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  30. 2021Detecting Dementia from Speech and Transcripts using Transformerscited by 13
  31. 2024Connected Speech-Based Cognitive Assessment in Chinese and Englishcited by 13
  32. 2023Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detectioncited by 11
  33. 2024ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasetscited by 11
  34. 2022Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speechcited by 10
  35. 2022Exploring linguistic feature and model combination for speech recognition based automatic AD detectioncited by 9
  36. 2023Cross-lingual Alzheimer's Disease detection based on paralinguistic and pre-trained featurescited by 8
  37. 2025CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detectioncited by 6
  38. 2023Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speechcited by 5

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.