# MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

arXiv: https://arxiv.org/abs/2411.12126 · HTML: https://mlsanity.com/paper/2411.12126/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 16 of 16)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification](https://mlsanity.com/paper/2002.04114/) (2020)
- [Mobile Sensor Data Anonymization](https://mlsanity.com/paper/1810.11546/) (2018)
- [FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring](https://mlsanity.com/paper/2012.07450/) (2020)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition](https://mlsanity.com/paper/2010.00734/) (2020)
- [Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck](https://mlsanity.com/paper/1908.07094/) (2019)
- [ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer's Disease](https://mlsanity.com/paper/2310.15301/) (2023)
- [Deep Learning for Scene Classification: A Survey](https://mlsanity.com/paper/2101.10531/) (2021)
- [Heteroskedastic Geospatial Tracking with Distributed Camera Networks](https://mlsanity.com/paper/2306.02407/) (2023)
- [SALSA: Semantically-Aware Latent Space Autoencoder](https://mlsanity.com/paper/2310.02744/) (2023)
- [GDTM: An Indoor Geospatial Tracking Dataset with Distributed Multimodal Sensors](https://mlsanity.com/paper/2402.14136/) (2024)

## Built on by (most cited first, 6 of 6)

- [MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series](https://mlsanity.com/paper/2509.25278/) (2025)
- [Foundation Models for CPS-IoT: Opportunities and Challenges](https://mlsanity.com/paper/2501.16368/) (2025)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [Tracking without Seeing: Geospatial Inference using Encrypted Traffic from Distributed Nodes](https://mlsanity.com/paper/2603.27811/) (2026)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals](https://mlsanity.com/paper/2605.18837/) (2026)

## Abstract (arXiv)

Multimodal sensing systems are increasingly prevalent in various real-world applications. Most existing multimodal learning approaches heavily rely on training with a large amount of synchronized, complete multimodal data. However, such a setting is impractical in real-world IoT sensing applications where data is typically collected by distributed nodes with heterogeneous data modalities, and is also rarely labeled. In this paper, we propose MMBind, a new data binding approach for multimodal learning on distributed and heterogeneous IoT data. The key idea of MMBind is to construct a pseudo-paired multimodal dataset for model training by binding data from disparate sources and incomplete modalities through a sufficiently descriptive shared modality. We also propose a weighted contrastive learning approach to handle domain shifts among disparate data, coupled with an adaptive multimodal learning architecture capable of training models with heterogeneous modality combinations. Evaluations on ten real-world multimodal datasets highlight that MMBind outperforms state-of-the-art baselines under varying degrees of data incompleteness and domain shift, and holds promise for advancing multimodal foundation model training in IoT applicationsfootnote (The source code is available via https://github.com/nesl/multimodal-bind).