# CogniVoice: Multimodal and Multilingual Fusion Networks for Mild Cognitive Impairment Assessment from Spontaneous Speech

arXiv: https://arxiv.org/abs/2407.13660 · HTML: https://mlsanity.com/paper/2407.13660/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [End-to-End Bias Mitigation by Modelling Biases in Corpora](https://mlsanity.com/paper/1909.06321/) (2019)
- [To BERT or Not To BERT: Comparing Speech and Language-based Approaches for Alzheimer's Disease Detection](https://mlsanity.com/paper/2008.01551/) (2020)
- [Toward Joint Language Modeling for Speech Units and Text](https://mlsanity.com/paper/2310.08715/) (2023)
- [Connected Speech-Based Cognitive Assessment in Chinese and English](https://mlsanity.com/paper/2406.10272/) (2024)
- [Cross-lingual Alzheimer's Disease detection based on paralinguistic and pre-trained features](https://mlsanity.com/paper/2303.07650/) (2023)
- [Enriching Complex Networks with Word Embeddings for Detecting Mild Cognitive Impairment from Speech Transcripts](https://mlsanity.com/paper/1704.08088/) (2017)
- [Cross-Lingual Transfer Learning for Alzheimer's Detection From Spontaneous Speech](https://mlsanity.com/paper/2303.03049/) (2023)
- [A Digital Language Coherence Marker for Monitoring Dementia](https://mlsanity.com/paper/2310.09623/) (2023)

## Built on by (most cited first, 8 of 8)

- [Speech Unlearning](https://mlsanity.com/paper/2506.00848/) (2025)
- [EqualizeIR: Mitigating Linguistic Biases in Retrieval Models](https://mlsanity.com/paper/2504.07115/) (2025)
- [Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods](https://mlsanity.com/paper/2510.26038/) (2025)
- [False positive bias in AI-powered speech-based cognitive screening for multilingual English speakers in the UK](https://mlsanity.com/paper/2602.13047/) (2026)
- [Unveil Multi-Picture Descriptions for Multilingual Mild Cognitive Impairment Detection via Contrastive Learning](https://mlsanity.com/paper/2505.17067/) (2025)
- [Enhancing and Exploring Mild Cognitive Impairment Detection with W2V-BERT-2.0](https://mlsanity.com/paper/2501.16201/) (2025)
- [FairFlow: Mitigating Dataset Biases through Undecided Learning](https://mlsanity.com/paper/2503.17632/) (2025)
- [Fair Cognitive Impairment Detection Through Unlearning](https://mlsanity.com/paper/2606.18571/) (2026)

## Abstract (arXiv)

Mild Cognitive Impairment (MCI) is a medical condition characterized by noticeable declines in memory and cognitive abilities, potentially affecting individual's daily activities. In this paper, we introduce CogniVoice, a novel multilingual and multimodal framework to detect MCI and estimate Mini-Mental State Examination (MMSE) scores by analyzing speech data and its textual transcriptions. The key component of CogniVoice is an ensemble multimodal and multilingual network based on ``Product of Experts'' that mitigates reliance on shortcut solutions. Using a comprehensive dataset containing both English and Chinese languages from TAUKADIAL challenge, CogniVoice outperforms the best performing baseline model on MCI classification and MMSE regression tasks by 2.8 and 4.1 points in F1 and RMSE respectively, and can effectively reduce the performance gap across different language groups by 0.7 points in F1.