mlsanityData to 14 Oct 2026

Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices

arXiv 2309.06612 ↗First posted 12 Sep 2023cs.LG cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

The recent surge of interest surrounding Multimodal Neural Networks (MM-NN) is attributed to their ability to effectively process and integrate multiscale information from diverse data sources. MM-NNs extract and fuse features from multiple modalities using adequate unimodal backbones and specific fusion networks. Although this helps strengthen the multimodal information representation, designing such networks is labor-intensive. It requires tuning the architectural parameters of the unimodal backbones, choosing the fusing point, and selecting the operations for fusion. Furthermore, multimodality AI is emerging as a cutting-edge option in Internet of Things (IoT) systems where inference latency and energy consumption are critical metrics in addition to accuracy. In this paper, we propose Harmonic-NAS, a framework for the joint optimization of unimodal backbones and multimodal fusion networks with hardware awareness on resource-constrained devices. Harmonic-NAS involves a two-tier optimization approach for the unimodal backbone architectures and fusion strategy and operators. By incorporating the hardware dimension into the optimization, evaluation results on various devices and multimodal datasets have demonstrated the superiority of Harmonic-NAS over state-of-the-art approaches achieving up to 10.9% accuracy improvement, 1.91x latency reduction, and 2.14x energy efficiency gain.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

29 of 44 references.

References · 29

  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  5. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  6. 2017Progressive Neural Architecture Searchcited by 487
  7. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  8. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  9. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  10. 2017Gated Multimodal Units for Information Fusioncited by 248
  11. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  12. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
17 more
  1. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  2. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  3. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  4. 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
  5. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 129
  6. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  7. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 86
  8. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  9. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 71
  10. 2019One-Shot Neural Architecture Search via Self-Evaluated Template Networkcited by 69
  11. 2021AlphaNet: Improved Training of Supernets with Alpha-Divergencecited by 36
  12. 2020Deep Multimodal Neural Architecture Searchcited by 34
  13. 2021MUFASA: Multimodal Fusion Architecture Search for Electronic Health Recordscited by 30
  14. 2022HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scalingcited by 14
  15. 2021BM-NAS: Bilevel Multimodal Neural Architecture Searchcited by 13
  16. 2023MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC Deploymentcited by 9
  17. 2022H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awarenesscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 4.

8 papers.

  1. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  2. 2025MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learningcited by 3
  3. 2024SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Searchcited by 1
  4. 2026Efficient Architecture Search under Leave-One-Subject-Out Evaluationcited by 1
  5. 2025Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systemscited by 0
  6. 2026HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Searchcited by 0
  7. 2026Efficient Leakage-Free Neural Architecture Search under Leave-One-Subject-Out Evaluationcited by 0
  8. 2026Deep Learning for Anomaly Detection in Railway Systems: A Structured Surveycited by 0