# CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding

arXiv: https://arxiv.org/abs/2401.09057 · HTML: https://mlsanity.com/paper/2401.09057/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 17 Jan 2024.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts](https://mlsanity.com/paper/2012.09165/) (2020)
- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Unsupervised Point Cloud Pre-Training via Occlusion Completion](https://mlsanity.com/paper/2010.01089/) (2020)
- [Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](https://mlsanity.com/paper/2109.00179/) (2021)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/1910.09165/) (2019)
- [Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting](https://mlsanity.com/paper/2302.13130/) (2023)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [Learning Physical Graph Representations from Visual Scenes](https://mlsanity.com/paper/2006.12373/) (2020)
- [P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding](https://mlsanity.com/paper/2012.13089/) (2020)

## Built on by (most cited first, 6 of 6)

- [SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation](https://mlsanity.com/paper/2502.13143/) (2025)
- [Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception](https://mlsanity.com/paper/2602.23069/) (2026)
- [Tri-Efficient Transfer Learning for Point Cloud Videos](https://mlsanity.com/paper/2606.24175/) (2026)
- [Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning](https://mlsanity.com/paper/2403.06289/) (2024)
- [Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning](https://mlsanity.com/paper/2607.00514/) (2026)
- [POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shaking](https://mlsanity.com/paper/2607.06649/) (2026)

## Abstract (arXiv)

This paper introduces a novel approach named CrossVideo, which aims to enhance self-supervised cross-modal contrastive learning in the field of point cloud video understanding. Traditional supervised learning methods encounter limitations due to data scarcity and challenges in label acquisition. To address these issues, we propose a self-supervised learning method that leverages the cross-modal relationship between point cloud videos and image videos to acquire meaningful feature representations. Intra-modal and cross-modal contrastive learning techniques are employed to facilitate effective comprehension of point cloud video. We also propose a multi-level contrastive approach for both modalities. Through extensive experiments, we demonstrate that our method significantly surpasses previous state-of-the-art approaches, and we conduct comprehensive ablation studies to validate the effectiveness of our proposed designs.