mlsanityData to 7 Oct 2026

Towards Realistic Scene Generation with LiDAR Diffusion Models

arXiv 2404.00815 ↗First posted 31 Mar 2024cs.CV cs.AI cs.ROcited by 67

On the map: World models → World modeling from LiDAR generation

also in 1 more
Abstract (arXiv)

Diffusion models (DMs) excel in photo-realistic image synthesis, but their adaptation to LiDAR scene generation poses a substantial hurdle. This is primarily because DMs operating in the point space struggle to preserve the curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of their representation power. In this paper, we propose LiDAR Diffusion Models (LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to capture the realism of LiDAR scenes by incorporating geometric priors into the learning pipeline. Our method targets three major desiderata: pattern realism, geometry realism, and object realism. Specifically, we introduce curve-wise compression to simulate real-world LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding for a full 3D object context. With these three core designs, our method achieves competitive performance on unconditional LiDAR generation in 64-beam scenario and state of the art on conditional LiDAR generation, while maintaining high efficiency compared to point-based DMs (up to 107× faster). Furthermore, by compressing LiDAR scenes into a latent space, we enable the controllability of DMs with various conditions such as semantic maps, camera views, and text prompts.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 63 of its 76 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

63 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learning to Generate Realistic LiDAR Point Clouds8 Sep

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  13. 2021Variational Diffusion Modelscited by 1,145
  14. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  15. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  16. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  17. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  18. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  19. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  20. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  21. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  22. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
  23. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  24. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  25. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 231
  26. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
  27. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 221
  28. 2021Projected GANs Converge Fastercited by 204
  29. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  30. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  31. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 165
  32. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  33. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  34. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  35. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  36. 2016Vehicle Detection from 3D Lidar Using Fully Convolutional Networkcited by 105
  37. 2023PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstructioncited by 84
  38. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 38; 2026: 18.

67 papers.

Sort:

By area:

  1. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  2. 20253D and 4D World Modeling: A Surveycited by 64
  3. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  4. 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenescited by 37
  5. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  6. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  7. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  8. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 14
  9. 2024X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarioscited by 13
  10. 2024Fast LiDAR Data Generation with Rectified Flowscited by 13
  11. 2025WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusioncited by 12
  12. 2024Diffusion Models in 3D Vision: A Surveycited by 11
38 more
  1. 2024Distilling Diffusion Models to Efficient 3D LiDAR Scene Completioncited by 10
  2. 2025SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understandingcited by 10
  3. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  4. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  5. 2025Instant4D: 4D Gaussian Splatting in Minutescited by 10
  6. 2024LOGen: Toward Lidar Object Generation by Point Diffusioncited by 9
  7. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
  8. 2025R2LDM: An Efficient 4D Radar Super-Resolution Framework Leveraging Diffusion Modelcited by 7
  9. 2025Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scenecited by 5
  10. 2025Towards Generating Realistic 3D Semantic Training Data for Autonomous Drivingcited by 5
  11. 2025Veila: Panoramic LiDAR Generation from a Monocular RGB Imagecited by 5
  12. 2025DiffPCN: Latent Diffusion Model Based on Multi-view Depth Images for Point Cloud Completioncited by 5
  13. 2025U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequencescited by 5
  14. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  15. 2024LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenescited by 4
  16. 2025MObI: Multimodal Object Inpainting Using Diffusion Modelscited by 4
  17. 2025Diffusion Distillation With Direct Preference Optimization For Efficient 3D LiDAR Scene Completioncited by 4
  18. 2025A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generationcited by 4
  19. 2026GenAssets: Generating in-the-wild 3D Assets in Latent Spacecited by 4
  20. 2025Generative Data Mining with Longtail-Guided Diffusioncited by 3
  21. 2025Industrial-Grade Sensor Simulation via Gaussian Splatting: A Modular Framework for Scalable Editing and Full-Stack Validationcited by 3
  22. 2025Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Modelscited by 2
  23. 2025Controllable Motion Generation via Diffusion Modal Couplingcited by 2
  24. 2025SG-LDM: Semantic-Guided LiDAR Generation via Latent-Aligned Diffusioncited by 2
  25. 2025Scene Coordinate Reconstruction Priorscited by 2
  26. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2
  27. 2025LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priorscited by 2
  28. 2025RadarGen: Automotive Radar Point Cloud Generation from Camerascited by 2
  29. 2026R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generationcited by 2
  30. 2026LiFlow: Flow Matching for 3D LiDAR Scene Completioncited by 2
  31. 2026L3DR: 3D-aware LiDAR Diffusion and Rectificationcited by 2
  32. 2026Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Drivingcited by 2
  33. 2024Map Imagination Like Blind Humans: Group Diffusion Model for Robotic Map Generationcited by 1
  34. 2025Physically Based Neural LiDAR Resimulationcited by 1
  35. 2025Range-Edit: Semantic Mask Guided Outdoor LiDAR Scene Editingcited by 1
  36. 2025LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputscited by 1
  37. 2026GEM: Generating LiDAR World Model via Deformable Mambacited by 1
  38. 2025MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priorscited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion
  2. 2023LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models
  3. 2024Simultaneous Diffusion Sampling for Conditional LiDAR Generation
  4. 2025LiDPM: Rethinking Point Diffusion for Lidar Scene Completion
  5. 2025Learning to Generate 4D LiDAR Sequences
  6. 2024RangeLDM: Fast Realistic LiDAR Point Cloud Generation
  7. 2024Fast LiDAR Upsampling using Conditional Diffusion Models
  8. 2026M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis