mlsanityData to 14 Oct 2026

Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

arXiv 2309.05300 ↗First posted 11 Sep 2023cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

The increasing availability of multi-sensor data sparks wide interest in multimodal self-supervised learning. However, most existing approaches learn only common representations across modalities while ignoring intra-modal training and modality-unique representations. We propose Decoupling Common and Unique Representations (DeCUR), a simple yet effective method for multimodal self-supervised learning. By distinguishing inter- and intra-modal embeddings through multimodal redundancy reduction, DeCUR can integrate complementary information across different modalities. We evaluate DeCUR in three common multimodal scenarios (radar-optical, RGB-elevation, and RGB-depth), and demonstrate its consistent improvement regardless of architectures and for both multimodal and modality-missing settings. With thorough experiments and comprehensive analysis, we hope this work can provide valuable insights and raise more interest in researching the hidden relationships of multimodal representations.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

46 of 64 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,384
34 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  9. 2017Large Batch Training of Convolutional Networkscited by 535
  10. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  12. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  13. 2022Vision Transformer with Deformable Attentioncited by 295
  14. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  15. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  16. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 215
  17. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  18. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  19. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  20. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  21. 2014Learning Rich Features from RGB-D Images for Object Detection and Segmentationcited by 127
  22. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  23. 2023Towards Geospatial Foundation Models via Continual Pretrainingcited by 109
  24. 2021BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrievalcited by 92
  25. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  26. 2022SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observationcited by 84
  27. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 77
  28. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  29. 2020Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentationcited by 48
  30. 2022Self-supervised Vision Transformers for Joint SAR-optical Representation Learningcited by 40
  31. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 32
  32. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  33. 2021ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognitioncited by 12
  34. 2023Feature Decomposition for Reducing Negative Transfer: A Novel Multi-task Learning Method for Recommender Systemcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 19; 2026: 11.

36 papers.

Sort:

By area:

  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 169
  2. 2024Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applicationscited by 144
  3. 2024MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretrainingcited by 66
  4. 2025Galileo: Learning Global & Local Features of Many Remote Sensing Modalitiescited by 43
  5. 2025TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysiscited by 39
  6. 2024On the Foundations of Earth and Climate Foundation Modelscited by 33
  7. 2025TerraTorch: The Geospatial Foundation Models Toolkitcited by 25
  8. 2025TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observationcited by 25
  9. 2025GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AIcited by 21
  10. 2024Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Spacecited by 12
  11. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  12. 2025GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Datacited by 8
24 more
  1. 2024CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentationcited by 7
  2. 2025DOFA-CLIP: Multimodal Vision-Language Foundation Models for Earth Observationcited by 7
  3. 2025Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendationcited by 6
  4. 2026Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Accesscited by 5
  5. 2025DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applicationscited by 4
  6. 2025GeoCrossBench: Cross-Band Generalization for Remote Sensingcited by 4
  7. 2025MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Datacited by 2
  8. 2026HighFM: Towards a Foundation Model for Learning Representations from High-Frequency Earth Observation Datacited by 2
  9. 2025Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representationcited by 1
  10. 2025HiRes-FusedMIM: A High-Resolution RGB-DSM Pre-trained Model for Building-Level Remote Sensing Applicationscited by 1
  11. 2025Assessing the value of Geo-Foundational Models for Flood Inundation Mapping: Benchmarking models for Sentinel-1, Sentinel-2, and Planetscope for end-userscited by 1
  12. 2026CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understandingcited by 1
  13. 2026Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencodercited by 1
  14. 2026FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scalescited by 1
  15. 2025Beyond H&E: Unlocking Pathological Insights with Polarization Imagingcited by 0
  16. 2025On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?cited by 0
  17. 2025A Plasticity-Aware Method for Continual Self-Supervised Learning in Remote Sensingcited by 0
  18. 2025A Deep-Learning Framework for Land-Sliding Classification from Remote Sensing Imagecited by 0
  19. 2025LandSegmenter: Towards a Flexible Foundation Model for Land Use and Land Cover Mappingcited by 0
  20. 2026Multitask Learning for Earth Observation Data Classification with Hybrid Quantum Networkcited by 0
  21. 2026Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learningcited by 0
  22. 2026HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learningcited by 0
  23. 2026CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrievalcited by 0
  24. 2026Toward Seasonal Guidelines for Robust Deep-Learning Sentinel-2 Building Detection in Different Area Typescited by 0