mlsanityData to 7 Oct 2026

Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks

arXiv 2110.04076 ↗First posted 28 Sep 2021cs.CV cs.ROcited by 42

On the map: World models → World modeling from LiDAR generation

Abstract (arXiv)

Exploiting past 3D LiDAR scans to predict future point clouds is a promising method for autonomous mobile systems to realize foresighted state estimation, collision avoidance, and planning. In this paper, we address the problem of predicting future 3D LiDAR point clouds given a sequence of past LiDAR scans. Estimating the future scene on the sensor level does not require any preceding steps as in localization or tracking systems and can be trained self-supervised. We propose an end-to-end approach that exploits a 2D range image representation of each 3D LiDAR scan and concatenates a sequence of range images to obtain a 3D tensor. Based on such tensors, we develop an encoder-decoder architecture using 3D convolutions to jointly aggregate spatial and temporal information of the scene and to predict the future 3D point clouds. We evaluate our method on multiple datasets and the experimental results suggest that our method outperforms existing point cloud prediction architectures and generalizes well to new, unseen environments without additional fine-tuning. Our method operates online and is faster than the common LiDAR frame rate of 10 Hz.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 28 of its 39 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

28 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space7 Jun

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  5. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  6. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  7. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  8. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  9. 2018FlowNet3D: Learning Scene Flow in 3D Point Cloudscited by 241
  10. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  11. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  12. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
15 more
  1. 2020MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Mapscited by 100
  2. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  3. 2018Deep Generative Modeling of LiDAR Datacited by 70
  4. 2021Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Datacited by 64
  5. 2019PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processingcited by 32
  6. 2017Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labelingcited by 30
  7. 2021Range Image-based LiDAR Localization for Autonomous Vehiclescited by 25
  8. 2018FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANscited by 24
  9. 2020LaserFlow: Efficient and Probabilistic Object Detection and Motion Forecastingcited by 22
  10. 2020MoNet: Motion-based Point Cloud Prediction Networkcited by 14
  11. 2020Double-Prong ConvLSTM for Spatiotemporal Occupancy Prediction in Dynamic Environmentscited by 13
  12. 2020PointINet: Point Cloud Frame Interpolation Networkcited by 12
  13. 2020RV-FuseNet: Range View Based Fusion of Time-Series LiDAR Data for Joint 3D Object Detection and Motion Forecastingcited by 8
  14. 2020Temporal LiDAR Frame Prediction for Autonomous Drivingcited by 7
  15. 20192D LiDAR Map Prediction via Estimating Motion Flow with GRUcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 16; 2024: 6; 2025: 10; 2026: 2.

42 papers.

Sort:

By area:

  1. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  2. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  3. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  4. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  5. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  6. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecastingcited by 76
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 2025A Survey of World Models for Autonomous Drivingcited by 60
  9. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
  10. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applicationscited by 50
  11. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 40
  12. 2022Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutionscited by 35
30 more
  1. 2023LiDAR-based 4D Occupancy Completion and Forecastingcited by 23
  2. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  3. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  4. 2022SeqOT: A Spatial-Temporal Transformer Network for Place Recognition Using Sequential LiDAR Datacited by 20
  5. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  6. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  7. 2023NeuralPCI: Spatio-temporal Neural Field for 3D Point Cloud Multi-frame Non-linear Interpolationcited by 13
  8. 2023PCPNet: An Efficient and Semantic-Enhanced Transformer Network for Point Cloud Predictioncited by 11
  9. 2025OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Frameworkcited by 11
  10. 2023STTracker: Spatio-Temporal Tracker for 3D Single Object Trackingcited by 10
  11. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  12. 20254D Visual Pre-training for Robot Learningcited by 8
  13. 2022Learning Constrained Dynamic Correlations in Spatiotemporal Graphs for Motion Predictioncited by 7
  14. 2024Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecastingcited by 6
  15. 2022Sequential Point Clouds: A Surveycited by 5
  16. 2023NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understandingcited by 5
  17. 2025Delta-Triplane Transformers as Occupancy World Modelscited by 5
  18. 2022HALS: A Height-Aware Lidar Super-Resolution Framework for Autonomous Drivingcited by 4
  19. 2022LOPR: Latent Occupancy PRediction using Generative Modelscited by 4
  20. 2022SeRP: Self-Supervised Representation Learning Using Perturbed Point Cloudscited by 3
  21. 2024Deep Bayesian Future Fusion for Self-Supervised, High-Resolution, Off-Road Mappingcited by 3
  22. 2023Self-Ordering Point Cloudscited by 2
  23. 2024Predicting Long-horizon Futures by Conditioning on Geometry and Timecited by 1
  24. 2025TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awarenesscited by 1
  25. 2026GEM: Generating LiDAR World Model via Deformable Mambacited by 1
  26. 2026MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Predictioncited by 1
  27. 2023Self-Supervised Learning for Point Clouds Data: A Surveycited by 0
  28. 2023Point2Point : A Framework for Efficient Deep Learning on Hilbert sorted Point Clouds with applications in Spatio-Temporal Occupancy Predictioncited by 0
  29. 2025Temporal Overlapping Prediction: A Self-supervised Pre-training Method for LiDAR Moving Object Segmentationcited by 0
  30. 2025LaGen: Towards Autoregressive LiDAR Scene Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Point Cloud Geometry as a Statistical Manifold: Theory and Practice ↗
  2. 2026R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation
  3. 2025Sonata: Self-Supervised Learning of Reliable Point Representations
  4. 2024RangeLDM: Fast Realistic LiDAR Point Cloud Generation
  5. 2026CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining
  6. 2026CloudDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain
  7. 2026Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds
  8. 2026On the Feasibility and Opportunity of Autoregressive 3D Object Detection