# Detecting Dementia from Speech and Transcripts using Transformers

arXiv: https://arxiv.org/abs/2110.14769 · HTML: https://mlsanity.com/paper/2110.14769/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 84 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [To BERT or Not To BERT: Comparing Speech and Language-based Approaches for Alzheimer's Disease Detection](https://mlsanity.com/paper/2008.01551/) (2020)
- [Alzheimer's Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs](https://mlsanity.com/paper/2106.15684/) (2021)
- [Comparing Acoustic-based Approaches for Alzheimer's Disease Detection](https://mlsanity.com/paper/2106.01555/) (2021)
- [Exploiting Multi-Modal Features From Pre-trained Networks for Alzheimer's Dementia Recognition](https://mlsanity.com/paper/2009.04070/) (2020)
- [Explainable Identification of Dementia from Transcripts using Transformer Networks](https://mlsanity.com/paper/2109.06980/) (2021)
- [auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1712.04382/) (2017)
- [Multimodal Inductive Transfer Learning for Detection of Alzheimer's Dementia and its Severity](https://mlsanity.com/paper/2009.00700/) (2020)
- [Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech](https://mlsanity.com/paper/2106.09668/) (2021)

## Built on by (most cited first, 13 of 13)

- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Voice Disorder Analysis: a Transformer-based Approach](https://mlsanity.com/paper/2406.14693/) (2024)
- [HAFFormer: A Hierarchical Attention-Free Framework for Alzheimer's Disease Detection From Spontaneous Speech](https://mlsanity.com/paper/2405.03952/) (2024)
- [Multimodal Detection of Bots on X (Twitter) using Transformers](https://mlsanity.com/paper/2308.14484/) (2023)
- [Depression detection in social media posts using transformer-based models and auxiliary features](https://mlsanity.com/paper/2409.20048/) (2024)
- [MVP: Multi-source Voice Pathology detection](https://mlsanity.com/paper/2505.20050/) (2025)
- [Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques](https://mlsanity.com/paper/2410.18972/) (2024)
- [Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening](https://mlsanity.com/paper/2606.14788/) (2026)
- [AI and Non AI Assessments for Dementia](https://mlsanity.com/paper/2307.01210/) (2023)
- [Dementia Insights: A Context-Based MultiModal Approach](https://mlsanity.com/paper/2503.01226/) (2025)
- [Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study](https://mlsanity.com/paper/2512.20948/) (2025)
- [A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning](https://mlsanity.com/paper/2605.25540/) (2026)

## Abstract (arXiv)

Alzheimer's disease (AD) constitutes a neurodegenerative disease with serious consequences to peoples' everyday lives, if it is not diagnosed early since there is no available cure. Alzheimer's is the most common cause of dementia, which constitutes a general term for loss of memory. Due to the fact that dementia affects speech, existing research initiatives focus on detecting dementia from spontaneous speech. However, little work has been done regarding the conversion of speech data to Log-Mel spectrograms and Mel-frequency cepstral coefficients (MFCCs) and the usage of pretrained models. Concurrently, little work has been done in terms of both the usage of transformer networks and the way the two modalities, i.e., speech and transcripts, are combined in a single neural network. To address these limitations, first we represent speech signal as an image and employ several pretrained models, with Vision Transformer (ViT) achieving the highest evaluation results. Secondly, we propose multimodal models. More specifically, our introduced models include Gated Multimodal Unit in order to control the influence of each modality towards the final classification and crossmodal attention so as to capture in an effective way the relationships between the two modalities. Extensive experiments conducted on the ADReSS Challenge dataset demonstrate the effectiveness of the proposed models and their superiority over state-of-the-art approaches.