# ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources

arXiv: https://arxiv.org/abs/2502.07862 · HTML: https://mlsanity.com/paper/2502.07862/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement](https://mlsanity.com/paper/1511.03995/) (2015)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning](https://mlsanity.com/paper/1904.05050/) (2019)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [Reparameterizable Subset Sampling via Continuous Relaxations](https://mlsanity.com/paper/1901.10517/) (2019)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Context-encoding Variational Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1812.05941/) (2018)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Heteroskedastic Geospatial Tracking with Distributed Camera Networks](https://mlsanity.com/paper/2306.02407/) (2023)
- [High Fidelity Image Synthesis With Deep VAEs In Latent Space](https://mlsanity.com/paper/2303.13714/) (2023)
- [GDTM: An Indoor Geospatial Tracking Dataset with Distributed Multimodal Sensors](https://mlsanity.com/paper/2402.14136/) (2024)

## Built on by (most cited first, 1 of 1)

- [SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations](https://mlsanity.com/paper/2604.26181/) (2026)

## Abstract (arXiv)

Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc.) and fluctuating quality of inputs (from sensor feed corruption, environmental noise, etc.). Statically provisioned multimodal systems cannot adapt when compute resources change over time, while existing dynamic networks struggle with strict compute budgets. Additionally, both systems often neglect the impact of variations in modality quality. Consequently, modalities suffering substantial corruption may needlessly consume resources better allocated towards other modalities. We propose ADMN, a layer-wise Adaptive Depth Multimodal Network capable of tackling both challenges: it adjusts the total number of active layers across all modalities to meet strict compute resource constraints and continually reallocates layers across input modalities according to their modality quality. Our evaluations showcase ADMN can match the accuracy of state-of-the-art networks while reducing up to 75% of their floating-point operations.