# Deep Multimodal Neural Architecture Search

arXiv: https://arxiv.org/abs/2004.12070 · HTML: https://mlsanity.com/paper/2004.12070/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration](https://mlsanity.com/paper/2108.07073/) (2021)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation](https://mlsanity.com/paper/2112.11691/) (2021)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [Revisiting Multimodal Fusion for 3D Anomaly Detection from an Architectural Perspective](https://mlsanity.com/paper/2412.17297/) (2024)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [Rationale-guided Prompting for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2412.16936/) (2024)
- [Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices](https://mlsanity.com/paper/2309.06612/) (2023)
- [Semantic-aware Modular Capsule Routing for Visual Question Answering](https://mlsanity.com/paper/2207.10404/) (2022)
- [Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems](https://mlsanity.com/paper/2104.04000/) (2021)
- [Bidirectional Contrastive Split Learning for Visual Question Answering](https://mlsanity.com/paper/2208.11435/) (2022)
- [Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA](https://mlsanity.com/paper/2308.15851/) (2023)
- [ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model](https://mlsanity.com/paper/2503.13938/) (2025)
- [Instance-Level Trojan Attacks on Visual Question Answering via Adversarial Learning in Neuron Activation Space](https://mlsanity.com/paper/2304.00436/) (2023)
- [Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2412.18351/) (2024)
- [Generalizable Prompt Learning of CLIP: A Brief Overview](https://mlsanity.com/paper/2503.01263/) (2025)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking](https://mlsanity.com/paper/2110.05159/) (2021)
- [Privacy Preserving Visual Question Answering](https://mlsanity.com/paper/2202.07712/) (2022)
- [Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering](https://mlsanity.com/paper/2203.12814/) (2022)
- [End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers](https://mlsanity.com/paper/2307.00873/) (2023)
- [DC and SA: Robust and Efficient Hyperparameter Optimization of Multi-subnetwork Deep Learning Models](https://mlsanity.com/paper/2202.11841/) (2022)

## Abstract (arXiv)

Designing effective neural networks is fundamentally important in deep multimodal learning. Most existing works focus on a single task and design neural architectures manually, which are highly task-specific and hard to generalize to different tasks. In this paper, we devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone, where each encoder or decoder block corresponds to an operation searched from a predefined operation pool. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks. By using a gradient-based NAS algorithm, the optimal architectures for different tasks are learned efficiently. Extensive ablation studies, comprehensive analysis, and comparative experimental results show that the obtained MMnasNet significantly outperforms existing state-of-the-art approaches across three multimodal learning tasks (over five datasets), including visual question answering, image-text matching, and visual grounding.