# Self-Supervised Pillar Motion Learning for Autonomous Driving

arXiv: https://arxiv.org/abs/2104.08683 · HTML: https://mlsanity.com/paper/2104.08683/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds](https://mlsanity.com/paper/1906.05332/) (2019)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Just Go with the Flow: Self-Supervised Scene Flow Estimation](https://mlsanity.com/paper/1912.00497/) (2019)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [Identifying Unknown Instances for Autonomous Driving](https://mlsanity.com/paper/1910.11296/) (2019)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction](https://mlsanity.com/paper/1811.11387/) (2018)
- [STEP: Spatio-Temporal Progressive Learning for Video Action Detection](https://mlsanity.com/paper/1904.09288/) (2019)

## Built on by (most cited first, 30 of 44)

- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Exploring Simple 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2108.10312/) (2021)
- [SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations](https://mlsanity.com/paper/2112.04680/) (2021)
- [PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2305.04925/) (2023)
- [ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals](https://mlsanity.com/paper/2303.12071/) (2023)
- [Dynamic 3D Scene Analysis by Point Cloud Accumulation](https://mlsanity.com/paper/2207.12394/) (2022)
- [DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2309.15109/) (2023)
- [TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2303.09998/) (2023)
- [DeFlow: Decoder of Scene Flow Network in Autonomous Driving](https://mlsanity.com/paper/2401.16122/) (2024)
- [IterativePFN: True Iterative Point Cloud Filtering](https://mlsanity.com/paper/2304.01529/) (2023)
- [GEDepth: Ground Embedding for Monocular Depth Estimation](https://mlsanity.com/paper/2309.09975/) (2023)
- [OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds](https://mlsanity.com/paper/2210.04458/) (2022)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [Self-Supervised 3D Scene Flow Estimation Guided by Superpoints](https://mlsanity.com/paper/2305.02528/) (2023)
- [360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View](https://mlsanity.com/paper/2303.11910/) (2023)
- [Contrastive Learning for Joint Normal Estimation and Point Cloud Filtering](https://mlsanity.com/paper/2208.06811/) (2022)
- [JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes](https://mlsanity.com/paper/2207.07895/) (2022)
- [Deep Transfer Learning for Intelligent Vehicle Perception: a Survey](https://mlsanity.com/paper/2306.15110/) (2023)
- [StraightPCF: Straight Point Cloud Filtering](https://mlsanity.com/paper/2405.08322/) (2024)
- [Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving](https://mlsanity.com/paper/2306.07276/) (2023)
- [Towards Cooperative Federated Learning over Heterogeneous Edge/Fog Networks](https://mlsanity.com/paper/2303.08361/) (2023)
- [ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks](https://mlsanity.com/paper/2111.13330/) (2021)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [Arbitrary Point Cloud Upsampling with Spherical Mixture of Gaussians](https://mlsanity.com/paper/2208.05274/) (2022)
- [Semi-Supervised Class-Agnostic Motion Prediction with Pseudo Label Regeneration and BEVMix](https://mlsanity.com/paper/2312.08009/) (2023)
- [LEGO-Motion: Learning-Enhanced Grids with Occupancy Instance Modeling for Class-Agnostic Motion Prediction](https://mlsanity.com/paper/2503.07367/) (2025)
- [Invisible Backdoor Attacks Using Data Poisoning in the Frequency Domain](https://mlsanity.com/paper/2207.04209/) (2022)
- [Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search](https://mlsanity.com/paper/2210.01384/) (2022)
- [Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals](https://mlsanity.com/paper/2401.11499/) (2024)
- [Self-Supervised Class-Agnostic Motion Prediction with Spatial and Temporal Consistency Regularizations](https://mlsanity.com/paper/2403.13261/) (2024)

## Abstract (arXiv)

Autonomous driving can benefit from motion behavior comprehension when interacting with diverse traffic participants in highly dynamic environments. Recently, there has been a growing interest in estimating class-agnostic motion directly from point clouds. Current motion estimation methods usually require vast amount of annotated training data from self-driving scenes. However, manually labeling point clouds is notoriously difficult, error-prone and time-consuming. In this paper, we seek to answer the research question of whether the abundant unlabeled data collections can be utilized for accurate and efficient motion learning. To this end, we propose a learning framework that leverages free supervisory signals from point clouds and paired camera images to estimate motion purely via self-supervision. Our model involves a point cloud based structural consistency augmented with probabilistic motion masking as well as a cross-sensor motion regularization to realize the desired self-supervision. Experiments reveal that our approach performs competitively to supervised methods, and achieves the state-of-the-art result when combining our self-supervised model with supervised fine-tuning.