# OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving arXiv: https://arxiv.org/abs/2412.17226 ยท HTML: https://mlsanity.com/paper/2412.17226/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in World modeling from LiDAR generation, one of 11 directions in World models. - Builds on 26 of its 42 references that are in the map. - First posted on arXiv 23 Dec 2024. ## Placed in - World models / [World modeling from LiDAR generation](https://mlsanity.com/wm/world-modeling-from-lidar-generation/) - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 26 of 26) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015) - [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018) - [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020) - [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021) - [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022) - [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021) - [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021) - [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022) - [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021) - [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023) - [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019) - [DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation](https://mlsanity.com/paper/2307.01831/) (2023) - [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021) - [UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation](https://mlsanity.com/paper/2311.01448/) (2023) - [Deep Generative Modeling of LiDAR Data](https://mlsanity.com/paper/1812.01180/) (2018) - [Weakly Supervised 3D Object Detection from Lidar Point Cloud](https://mlsanity.com/paper/2007.11901/) (2020) - [Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis](https://mlsanity.com/paper/2204.06307/) (2022) - [Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving](https://mlsanity.com/paper/2305.15765/) (2023) - [Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency](https://mlsanity.com/paper/2303.08686/) (2023) - [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022) ## Built on by (most cited first, 13 of 13) - [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025) - [LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences](https://mlsanity.com/paper/2508.03692/) (2025) - [AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models](https://mlsanity.com/paper/2511.20325/) (2025) - [RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation](https://mlsanity.com/paper/2509.16500/) (2025) - [CausalDrive: Real-time Causal World Models for Autonomous Driving](https://mlsanity.com/paper/2606.15341/) (2026) - [Diffusion Distillation With Direct Preference Optimization For Efficient 3D LiDAR Scene Completion](https://mlsanity.com/paper/2504.11447/) (2025) - [L3DR: 3D-aware LiDAR Diffusion and Rectification](https://mlsanity.com/paper/2602.19064/) (2026) - [DriveLiDAR4D: Sequential and Controllable LiDAR Scene Generation for Autonomous Driving](https://mlsanity.com/paper/2511.13309/) (2025) - [R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation](https://mlsanity.com/paper/2601.07692/) (2026) - [Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detection](https://mlsanity.com/paper/2503.08992/) (2025) - [Breaking Down Monocular Ambiguity: Exploiting Temporal Evolution for 3D Lane Detection](https://mlsanity.com/paper/2504.20525/) (2025) - [PointSlice: Accurate and Efficient Slice-Based Representation for 3D Object Detection from Point Clouds](https://mlsanity.com/paper/2509.01487/) (2025) - [CoLC: Communication-Efficient Collaborative Perception with LiDAR Completion](https://mlsanity.com/paper/2603.00682/) (2026) ## Abstract (arXiv) To enhance autonomous driving safety in complex scenarios, various methods have been proposed to simulate LiDAR point cloud data. Nevertheless, these methods often face challenges in producing high-quality, diverse, and controllable foreground objects. To address the needs of object-aware tasks in 3D perception, we introduce OLiDM, a novel framework capable of generating high-fidelity LiDAR data at both the object and the scene levels. OLiDM consists of two pivotal components: the Object-Scene Progressive Generation (OPG) module and the Object Semantic Alignment (OSA) module. OPG adapts to user-specific prompts to generate desired foreground objects, which are subsequently employed as conditions in scene generation, ensuring controllable outputs at both the object and scene levels. This also facilitates the association of user-defined object-level annotations with the generated LiDAR scenes. Moreover, OSA aims to rectify the misalignment between foreground objects and background scenes, enhancing the overall quality of the generated objects. The broad effectiveness of OLiDM is demonstrated across various LiDAR generation tasks, as well as in 3D perception tasks. Specifically, on the KITTI-360 dataset, OLiDM surpasses prior state-of-the-art methods such as UltraLiDAR by 17.5 in FPD. Additionally, in sparse-to-dense LiDAR completion, OLiDM achieves a significant improvement over LiDARGen, with a 57.47% increase in semantic IoU. Moreover, OLiDM enhances the performance of mainstream 3D detectors by 2.4% in mAP and 1.9% in NDS, underscoring its potential in advancing object-aware 3D tasks. Code is available at: https://yanty123.github.io/OLiDM.