# Cross and Learn: Cross-Modal Self-Supervision

arXiv: https://arxiv.org/abs/1811.03879 · HTML: https://mlsanity.com/paper/1811.03879/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2018.

## Built on by (most cited first, 30 of 34)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Home Action Genome: Cooperative Compositional Action Understanding](https://mlsanity.com/paper/2105.05226/) (2021)
- [Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentation](https://mlsanity.com/paper/2101.07253/) (2021)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.02618/) (2021)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks](https://mlsanity.com/paper/2104.07917/) (2021)
- [Self-supervised Transformer for Deepfake Detection](https://mlsanity.com/paper/2203.01265/) (2022)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Motion-Augmented Self-Training for Video Recognition at Smaller Scale](https://mlsanity.com/paper/2105.01646/) (2021)
- [Cross-Modal Knowledge Transfer Without Task-Relevant Source Data](https://mlsanity.com/paper/2209.04027/) (2022)
- [A Survey of Generative Techniques for Spatial-Temporal Data Mining](https://mlsanity.com/paper/2405.09592/) (2024)
- [CoCon: Cooperative-Contrastive Learning](https://mlsanity.com/paper/2104.14764/) (2021)
- [Vision transformers in domain adaptation and domain generalization: a study of robustness](https://mlsanity.com/paper/2404.04452/) (2024)
- [MaCLR: Motion-aware Contrastive Learning of Representations for Videos](https://mlsanity.com/paper/2106.09703/) (2021)
- [SS-3DCapsNet: Self-supervised 3D Capsule Networks for Medical Segmentation on Less Labeled Data](https://mlsanity.com/paper/2201.05905/) (2022)
- [Prospective Role of Foundation Models in Advancing Autonomous Vehicles](https://mlsanity.com/paper/2405.02288/) (2023)
- [Self-Supervised Learning via multi-Transformation Classification for Action Recognition](https://mlsanity.com/paper/2102.10378/) (2021)
- [Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity](https://mlsanity.com/paper/2209.07811/) (2022)
- [A Large-Scale Analysis on Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2306.06010/) (2023)
- [Going Deeper into Recognizing Actions in Dark Environments: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2202.09545/) (2022)
- [Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data](https://mlsanity.com/paper/2401.05014/) (2024)
- [Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations](https://mlsanity.com/paper/2103.00787/) (2021)
- [Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised Learning](https://mlsanity.com/paper/2210.06297/) (2022)
- [A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2504.06153/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Cross-modal Image Retrieval with Deep Mutual Information Maximization](https://mlsanity.com/paper/2103.06032/) (2021)
- [PreViTS: Contrastive Pretraining with Video Tracking Supervision](https://mlsanity.com/paper/2112.00804/) (2021)
- [Information Theory-Guided Heuristic Progressive Multi-View Coding](https://mlsanity.com/paper/2109.02344/) (2021)
- [Feature-Supervised Action Modality Transfer](https://mlsanity.com/paper/2108.03329/) (2021)

## Abstract (arXiv)

In this paper we present a self-supervised method for representation learning utilizing two different modalities. Based on the observation that cross-modal information has a high semantic meaning we propose a method to effectively exploit this signal. For our approach we utilize video data since it is available on a large scale and provides easily accessible modalities given by RGB and optical flow. We demonstrate state-of-the-art performance on highly contested action recognition datasets in the context of self-supervised learning. We show that our feature representation also transfers to other tasks and conduct extensive ablation studies to validate our core contributions. Code and model can be found at https://github.com/nawidsayed/Cross-and-Learn.