# Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

arXiv: https://arxiv.org/abs/2302.05894 · HTML: https://mlsanity.com/paper/2302.05894/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering](https://mlsanity.com/paper/1708.03619/) (2017)
- [BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection](https://mlsanity.com/paper/1902.00038/) (2019)
- [Explainable Identification of Dementia from Transcripts using Transformer Networks](https://mlsanity.com/paper/2109.06980/) (2021)
- [Multimodal Inductive Transfer Learning for Detection of Alzheimer's Dementia and its Severity](https://mlsanity.com/paper/2009.00700/) (2020)
- [The INESC-ID Multi-Modal System for the ADReSS 2020 Challenge](https://mlsanity.com/paper/2005.14646/) (2020)
- [Graph-based Neural Architecture Search with Operation Embeddings](https://mlsanity.com/paper/2105.04885/) (2021)

## Built on by (most cited first, 4 of 4)

- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Breaking Data Efficiency Dilemma: A Federated and Augmented Learning Framework For Alzheimer's Disease Detection via Speech](https://mlsanity.com/paper/2602.14655/) (2026)
- [A Graph Neural Architecture Search Approach for Identifying Bots in Social Media](https://mlsanity.com/paper/2411.16285/) (2024)
- [LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection](https://mlsanity.com/paper/2609.10896/) (2026)

## Abstract (arXiv)

Alzheimer's dementia (AD) affects memory, thinking, and language, deteriorating person's life. An early diagnosis is very important as it enables the person to receive medical help and ensure quality of life. Therefore, leveraging spontaneous speech in conjunction with machine learning methods for recognizing AD patients has emerged into a hot topic. Most of the previous works employ Convolutional Neural Networks (CNNs), to process the input signal. However, finding a CNN architecture is a time-consuming process and requires domain expertise. Moreover, the researchers introduce early and late fusion approaches for fusing different modalities or concatenate the representations of the different modalities during training, thus the inter-modal interactions are not captured. To tackle these limitations, first we exploit a Neural Architecture Search (NAS) method to automatically find a high performing CNN architecture. Next, we exploit several fusion methods, including Multimodal Factorized Bilinear Pooling and Tucker Decomposition, to combine both speech and text modalities. To the best of our knowledge, there is no prior work exploiting a NAS approach and these fusion methods in the task of dementia detection from spontaneous speech. We perform extensive experiments on the ADReSS Challenge dataset and show the effectiveness of our approach over state-of-the-art methods.