InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals
Not on the map. Why?
Abstract (arXiv)
Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves textitefficient cross-modal alignment with textitlimited data pairs through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 50 of its 81 references that are in the map. First posted on arXiv 13 Apr 2025.
Builds on
50 of 81 references.
References · 50
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2019Contrastive Multiview Codingcited by 1,301
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2018Disentangling by Factorisingcited by 616
- 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
38 more
- 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
- 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 345
- 2022Masked Autoencoders that Listencited by 273
- 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
- 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 250
- 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
- 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
- 2021Unsupervised Representation Learning for Time Series with Temporal Neighborhood Codingcited by 224
- 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
- 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
- 2018Learning Factorized Multimodal Representationscited by 171
- 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
- 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
- 2022Contrastive Audio-Visual Masked Autoencodercited by 146
- 20234M: Massively Multimodal Masked Modelingcited by 120
- 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
- 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
- 2021Robust Audio-Visual Instance Discriminationcited by 80
- 2023Siamese Masked Autoencoderscited by 76
- 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
- 2021Reconsidering Representation Alignment for Multi-view Clusteringcited by 57
- 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
- 2022Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representationscited by 48
- 2020CoMIR: Contrastive Multimodal Image Representation for Registrationcited by 45
- 2022Robust Contrastive Learning against Noisy Viewscited by 45
- 2022SimPer: Simple Self-Supervised Learning of Periodic Targetscited by 44
- 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
- 2022Geometric Multimodal Contrastive Representation Learningcited by 40
- 2022Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classificationcited by 39
- 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
- 2023Connecting Multi-modal Contrastive Representationscited by 36
- 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 33
- 2022COCOA: Cross Modality Contrastive Learning for Sensor Datacited by 21
- 2018Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Datacited by 19
- 2022Contrastive Masked Autoencoders for Self-Supervised Video Hashingcited by 13
- 2022Gacs-Korner Common Information Variational Autoencodercited by 12
- 2022Multimodal sensor fusion in the latent representation spacecited by 9
- 2019Learning Affective Correspondence between Music and Imagecited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.