# Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals

arXiv: https://arxiv.org/abs/2401.11499 · HTML: https://mlsanity.com/paper/2401.11499/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 21 Jan 2024.

## Builds on (most cited first, 30 of 38)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [FLOT: Scene Flow on Point Clouds Guided by Optimal Transport](https://mlsanity.com/paper/2007.11142/) (2020)
- [HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds](https://mlsanity.com/paper/1906.05332/) (2019)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Just Go with the Flow: Self-Supervised Scene Flow Estimation](https://mlsanity.com/paper/1912.00497/) (2019)
- [Neural Scene Flow Prior](https://mlsanity.com/paper/2111.01253/) (2021)
- [Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting](https://mlsanity.com/paper/2302.13130/) (2023)
- [Identifying Unknown Instances for Autonomous Driving](https://mlsanity.com/paper/1910.11296/) (2019)
- [FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation](https://mlsanity.com/paper/2011.10147/) (2020)
- [3D Point Cloud Processing and Learning for Autonomous Driving](https://mlsanity.com/paper/2003.00601/) (2020)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow](https://mlsanity.com/paper/2309.16940/) (2023)
- [CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2111.10502/) (2021)
- [Scalable Scene Flow from Point Clouds in the Real World](https://mlsanity.com/paper/2103.01306/) (2021)
- [Scene Flow from Point Clouds with or without Learning](https://mlsanity.com/paper/2011.00320/) (2020)
- [Self-Supervised Pillar Motion Learning for Autonomous Driving](https://mlsanity.com/paper/2104.08683/) (2021)
- [Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion](https://mlsanity.com/paper/2009.10467/) (2020)
- [PointPWC-Net: A Coarse-to-Fine Network for Supervised and Self-Supervised Scene Flow Estimation on 3D Point Clouds](https://mlsanity.com/paper/1911.12408/) (2019)

## Built on by (most cited first, 5 of 5)

- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [LEGO-Motion: Learning-Enhanced Grids with Occupancy Instance Modeling for Class-Agnostic Motion Prediction](https://mlsanity.com/paper/2503.07367/) (2025)
- [FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units](https://mlsanity.com/paper/2401.12862/) (2024)
- [Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2509.13116/) (2025)
- [Long-Term 3D Point Tracking By Cost Volume Fusion](https://mlsanity.com/paper/2407.13337/) (2024)

## Abstract (arXiv)

Learning the dense bird's eye view (BEV) motion flow in a self-supervised manner is an emerging research for robotics and autonomous driving. Current self-supervised methods mainly rely on point correspondences between point clouds, which may introduce the problems of fake flow and inconsistency, hindering the model's ability to learn accurate and realistic motion. In this paper, we introduce a novel cross-modality self-supervised training framework that effectively addresses these issues by leveraging multi-modality data to obtain supervision signals. We design three innovative supervision signals to preserve the inherent properties of scene motion, including the masked Chamfer distance loss, the piecewise rigidity loss, and the temporal consistency loss. Through extensive experiments, we demonstrate that our proposed self-supervised framework outperforms all previous self-supervision methods for the motion prediction task.