# A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer

arXiv: https://arxiv.org/abs/2211.04368 · HTML: https://mlsanity.com/paper/2211.04368/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Detecting cognitive decline using speech only: The ADReSSo Challenge](https://mlsanity.com/paper/2104.09356/) (2021)
- [Exploiting Multi-Modal Features From Pre-trained Networks for Alzheimer's Dementia Recognition](https://mlsanity.com/paper/2009.04070/) (2020)
- [Multimodal Inductive Transfer Learning for Detection of Alzheimer's Dementia and its Severity](https://mlsanity.com/paper/2009.00700/) (2020)
- [Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech](https://mlsanity.com/paper/2106.09668/) (2021)
- [The INESC-ID Multi-Modal System for the ADReSS 2020 Challenge](https://mlsanity.com/paper/2005.14646/) (2020)
- [Multimodal Unified Attention Networks for Vision-and-Language Interactions](https://mlsanity.com/paper/1908.04107/) (2019)
- [Multi-Modal Detection of Alzheimer's Disease from Speech and Text](https://mlsanity.com/paper/2012.00096/) (2020)

## Built on by (most cited first, 6 of 6)

- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Depression detection in social media posts using transformer-based models and auxiliary features](https://mlsanity.com/paper/2409.20048/) (2024)
- [Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques](https://mlsanity.com/paper/2410.18972/) (2024)
- [A Graph Neural Architecture Search Approach for Identifying Bots in Social Media](https://mlsanity.com/paper/2411.16285/) (2024)
- [A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning](https://mlsanity.com/paper/2605.25540/) (2026)
- [Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection](https://mlsanity.com/paper/2606.30675/) (2026)

## Abstract (arXiv)

Alzheimer's disease (AD) is a progressive neurological disorder, meaning that the symptoms develop gradually throughout the years. It is also the main cause of dementia, which affects memory, thinking skills, and mental abilities. Nowadays, researchers have moved their interest towards AD detection from spontaneous speech, since it constitutes a time-effective procedure. However, existing state-of-the-art works proposing multimodal approaches do not take into consideration the inter- and intra-modal interactions and propose early and late fusion approaches. To tackle these limitations, we propose deep neural networks, which can be trained in an end-to-end trainable way and capture the inter- and intra-modal interactions. Firstly, each audio file is converted to an image consisting of three channels, i.e., log-Mel spectrogram, delta, and delta-delta. Next, each transcript is passed through a BERT model followed by a gated self-attention layer. Similarly, each image is passed through a Swin Transformer followed by an independent gated self-attention layer. Acoustic features are extracted also from each audio file. Finally, the representation vectors from the different modalities are fed to a tensor fusion layer for capturing the inter-modal interactions. Extensive experiments conducted on the ADReSS Challenge dataset indicate that our introduced approaches obtain valuable advantages over existing research initiatives reaching Accuracy and F1-score up to 86.25% and 85.48% respectively.