# Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

arXiv: https://arxiv.org/abs/2607.00514 · HTML: https://mlsanity.com/paper/2607.00514/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 46 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 30 of 40)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](https://mlsanity.com/paper/2109.00179/) (2021)
- [Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining](https://mlsanity.com/paper/2302.02318/) (2023)
- [Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders](https://mlsanity.com/paper/2212.06785/) (2022)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?](https://mlsanity.com/paper/2212.08320/) (2022)
- [Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos](https://mlsanity.com/paper/2412.09621/) (2024)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations](https://mlsanity.com/paper/2510.23607/) (2025)
- [Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding](https://mlsanity.com/paper/2208.00281/) (2022)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [Point-JEPA: A Joint Embedding Predictive Architecture for Self-Supervised Learning on Point Cloud](https://mlsanity.com/paper/2404.16432/) (2024)
- [PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2305.04075/) (2023)

## Built on by (most cited first, 1 of 1)

- [Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos](https://mlsanity.com/paper/2608.24093/) (2026)

## Abstract (arXiv)

Automatic understanding of dynamic 4D point clouds, the 3D-point sequences captured over time by depth sensors and LiDAR, is central to robotics and embodied perception. Yet annotating them densely is expensive, making self-supervised pretraining the natural route to transferable representations. Existing pretext tasks, however, are almost entirely intra-modal, and the few methods that transfer knowledge from 2D foundation models rely on a single global embedding per clip, discarding the rich per-patch semantics that these models compute. To address this gap, we propose Cross4D-JEPA, a teacher-student method that distills a frozen 2D foundation model, an image model DINOv2, or a video model V-JEPA 2, into a 4D point encoder. The proposed method combines (1) a dense cross-modal correspondence that maps every 3D point to the teacher patch feature it projects to, and (2) a per-point objective that trains the student to match these features in latent space with no masking, negatives, or decoder. We evaluate Cross4D-JEPA on four benchmarks, MSR-Action3D, DeformingThings4D, NTU-RGB+D 60, and HOI4D, against intra-modal and global cross-modal baselines. Experimental results show that, under a matched protocol, the proposed method consistently outperforms intra-modal and global cross-modal baselines across the four benchmarks and is competitive with heavier published 4D methods; further analysis attributes this gain primarily to the granularity of the correspondence rather than the teacher modality. Beyond recognition accuracy, the dense representation learned by Cross4D-JEPA transfers across domains, improves label efficiency, and improves full-label fine-tuning under the same training budget, while a 13x smaller encoder matches a heavyweight pooling backbone.