# Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving

arXiv: https://arxiv.org/abs/2210.08061 · HTML: https://mlsanity.com/paper/2210.08061/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 86 of its 116 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 30 of 86)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)

## Built on by (most cited first, 30 of 59)

- [STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes](https://mlsanity.com/paper/2501.00602/) (2024)
- [Better Call SAL: Towards Learning to Segment Anything in Lidar](https://mlsanity.com/paper/2403.13129/) (2024)
- [Fast Neural Scene Flow](https://mlsanity.com/paper/2304.09121/) (2023)
- [SeFlow: A Self-Supervised Scene Flow Method in Autonomous Driving](https://mlsanity.com/paper/2407.01702/) (2024)
- [Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving](https://mlsanity.com/paper/2309.14491/) (2023)
- [ICP-Flow: LiDAR Scene Flow Estimation with ICP](https://mlsanity.com/paper/2402.17351/) (2024)
- [DeFlow: Decoder of Scene Flow Network in Autonomous Driving](https://mlsanity.com/paper/2401.16122/) (2024)
- [Towards Unsupervised Object Detection From LiDAR Point Clouds](https://mlsanity.com/paper/2311.02007/) (2023)
- [I Can't Believe It's Not Scene Flow!](https://mlsanity.com/paper/2403.04739/) (2024)
- [LISO: Lidar-only Self-Supervised 3D Object Detection](https://mlsanity.com/paper/2403.07071/) (2024)
- [UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes](https://mlsanity.com/paper/2405.15688/) (2024)
- [MV-Map: Offboard HD-Map Generation with Multi-view Consistency](https://mlsanity.com/paper/2305.08851/) (2023)
- [Re-Evaluating LiDAR Scene Flow for Autonomous Driving](https://mlsanity.com/paper/2304.02150/) (2023)
- [3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling](https://mlsanity.com/paper/2402.18146/) (2024)
- [Neural Eulerian Scene Flow Fields](https://mlsanity.com/paper/2410.02031/) (2024)
- [ZeroFlow: Scalable Scene Flow via Distillation](https://mlsanity.com/paper/2305.10424/) (2023)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [SeMoLi: What Moves Together Belongs Together](https://mlsanity.com/paper/2402.19463/) (2024)
- [SSF: Sparse Long-Range Scene Flow for Autonomous Driving](https://mlsanity.com/paper/2501.17821/) (2025)
- [Multi-Body Neural Scene Flow](https://mlsanity.com/paper/2310.10301/) (2023)
- [Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving](https://mlsanity.com/paper/2403.08504/) (2024)
- [HiMo: High-Speed Objects Motion Compensation in Point Clouds](https://mlsanity.com/paper/2503.00803/) (2025)
- [DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Diffusion Model](https://mlsanity.com/paper/2311.17456/) (2023)
- [VoteFlow: Enforcing Local Rigidity in Self-Supervised Scene Flow](https://mlsanity.com/paper/2503.22328/) (2025)
- [Floxels: Fast Unsupervised Voxel Based Scene Flow Estimation](https://mlsanity.com/paper/2503.04718/) (2025)
- [Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection](https://mlsanity.com/paper/2406.10115/) (2024)
- [Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detection](https://mlsanity.com/paper/2408.00619/) (2024)
- [Lidar Panoptic Segmentation in an Open World](https://mlsanity.com/paper/2409.14273/) (2024)
- [DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method](https://mlsanity.com/paper/2508.17054/) (2025)
- [A Cross-Scale Hierarchical Transformer with Correspondence-Augmented Attention for inferring Bird's-Eye-View Semantic Segmentation](https://mlsanity.com/paper/2304.03650/) (2023)

## Abstract (arXiv)

Learning-based perception and prediction modules in modern autonomous driving systems typically rely on expensive human annotation and are designed to perceive only a handful of predefined object categories. This closed-set paradigm is insufficient for the safety-critical autonomous driving task, where the autonomous vehicle needs to process arbitrarily many types of traffic participants and their motion behaviors in a highly dynamic world. To address this difficulty, this paper pioneers a novel and challenging direction, i.e., training perception and prediction models to understand open-set moving objects, with no human supervision. Our proposed framework uses self-learned flow to trigger an automated meta labeling pipeline to achieve automatic supervision. 3D detection experiments on the Waymo Open Dataset show that our method significantly outperforms classical unsupervised approaches and is even competitive to the counterpart with supervised scene flow. We further show that our approach generates highly promising results in open-set 3D detection and trajectory prediction, confirming its potential in closing the safety gap of fully supervised systems.