mlsanityData to 7 Oct 2026

Learning Aligned Cross-Modal Representations from Weakly Aligned Data

arXiv 1607.07295 ↗First posted 25 Jul 2016cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize cross-modal scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for cross-modal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 10; 2023: 8; 2024: 6; 2025: 1; 2026: 2.

34 papers.

Sort:

By area:

  1. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  2. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  3. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  4. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  5. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  6. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  7. 2021Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentationcited by 40
  8. 2021Spectral Unsupervised Domain Adaptation for Visual Recognitioncited by 40
  9. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  10. 2023Lossless Adaptation of Pretrained Vision Models For Robotic Manipulationcited by 28
  11. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  12. 2021Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Datacited by 10
22 more
  1. 2023Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedbackcited by 10
  2. 2022Sound and Visual Representation Learning with Multiple Pretraining Taskscited by 5
  3. 2021Template-Free Try-on Image Synthesis via Semantic-guided Optimizationcited by 4
  4. 2021Machine Learning in Nuclear Physicscited by 4
  5. 2022Cross-Modal Alignment Learning of Vision-Language Conceptual Systemscited by 4
  6. 2023Training Transitive and Commutative Multimodal Transformers with LoReTTacited by 4
  7. 2023A Client-server Deep Federated Learning for Cross-domain Surgical Image Segmentationcited by 4
  8. 2024Seeing Faces in Things: A Model and Dataset for Pareidoliacited by 4
  9. 2025OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentationcited by 4
  10. 2023M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understandingcited by 2
  11. 2024Domain Adaptation for Large-Vocabulary Object Detectorscited by 2
  12. 2021Information Theory-Guided Heuristic Progressive Multi-View Codingcited by 1
  13. 2023SF-FSDA: Source-Free Few-Shot Domain Adaptive Object Detection with Efficient Labeled Data Factorycited by 1
  14. 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneitycited by 1
  15. 2024Artificial Intelligence for Geometry-Based Feature Extraction, Analysis and Synthesis in Artistic Images: A Surveycited by 1
  16. 2022Weakly-Supervised Action Detection Guided by Audio Narrationcited by 0
  17. 2022Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokenscited by 0
  18. 2022Unifying Tracking and Image-Video Object Detectioncited by 0
  19. 2023Information Theory-Guided Heuristic Progressive Multi-View Codingcited by 0
  20. 2024A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoderscited by 0
  21. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0
  22. 2026Multi-Modal Guided Multi-Source Domain Adaptation for Object Detectioncited by 0