# Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech

arXiv: https://arxiv.org/abs/2106.09668 · HTML: https://mlsanity.com/paper/2106.09668/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 25 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 3 of 3)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)

## Built on by (most cited first, 16 of 16)

- [Alzheimer's Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs](https://mlsanity.com/paper/2106.15684/) (2021)
- [Explainable Identification of Dementia from Transcripts using Transformer Networks](https://mlsanity.com/paper/2109.06980/) (2021)
- [Detecting Dementia from Speech and Transcripts using Transformers](https://mlsanity.com/paper/2110.14769/) (2021)
- [A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer](https://mlsanity.com/paper/2211.04368/) (2022)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses](https://mlsanity.com/paper/2306.03443/) (2023)
- [A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection](https://mlsanity.com/paper/2410.19898/) (2024)
- [Alzheimers Dementia Detection using Acoustic & Linguistic features and Pre-Trained BERT](https://mlsanity.com/paper/2109.11010/) (2021)
- [Linguistic-Based Mild Cognitive Impairment Detection Using Informative Loss](https://mlsanity.com/paper/2402.01690/) (2024)
- [Influence of ASR and Language Model on Alzheimer's Disease Detection](https://mlsanity.com/paper/2110.15704/) (2021)
- [Zero-Shot Cognitive Impairment Detection from Speech Using AudioLLM](https://mlsanity.com/paper/2506.17351/) (2025)
- [Multimodal Audio-based Disease Prediction with Transformer-based Hierarchical Fusion Network](https://mlsanity.com/paper/2410.09289/) (2024)
- [Developing Conversational Speech Systems for Robots to Detect Speech Biomarkers of Cognition in People Living with Dementia](https://mlsanity.com/paper/2502.10896/) (2025)
- [Dementia classification from spontaneous speech using wrapper-based feature selection](https://mlsanity.com/paper/2502.03484/) (2025)
- [A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning](https://mlsanity.com/paper/2605.25540/) (2026)
- [Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models](https://mlsanity.com/paper/2607.21496/) (2026)

## Abstract (arXiv)

This paper is a submission to the Alzheimer's Dementia Recognition through Spontaneous Speech (ADReSS) challenge, which aims to develop methods that can assist in the automated prediction of severity of Alzheimer's Disease from speech data. We focus on acoustic and natural language features for cognitive impairment detection in spontaneous speech in the context of Alzheimer's Disease Diagnosis and the mini-mental state examination (MMSE) score prediction. We proposed a model that obtains unimodal decisions from different LSTMs, one for each modality of text and audio, and then combines them using a gating mechanism for the final prediction. We focused on sequential modelling of text and audio and investigated whether the disfluencies present in individuals' speech relate to the extent of their cognitive impairment. Our results show that the proposed classification and regression schemes obtain very promising results on both development and test sets. This suggests Alzheimer's Disease can be detected successfully with sequence modeling of the speech data of medical sessions.