GEM: Generating LiDAR World Model via Deformable Mamba
On the map: World models → World modeling from LiDAR generation
Abstract (arXiv)
World models, which simulate environmental dynamics and generate sensor observations, are gaining increasing attention in autonomous driving. However, progress in LiDAR-based world models has lagged behind those built on camera videos or occupancy data, primarily due to two core challenges: the inherent disorder of LiDAR point clouds and the difficulty of distinguishing dynamic objects from static structures. To address these issues, we propose GEM: a Generative LiDAR world model that leverages deformable mamba architecture, significantly improving fidelity and imaginative capability. Specifically, leveraging the structural similarity between sequential laser scanning and Mamba's processing mechanism, we first tokenize LiDAR sweeps into compact representations via a custom LiDAR scene tokenizer. After unsupervised disentanglement of tokenized features via a dynamic-static separator, a tri-path deformable Mamba is introduced to perform selective scanning and adaptive gating fusion over the disentangled features, leading to enhanced spatial-temporal understanding of the world evolution. Optionally, a planner and a BEV layout controller can be integrated to explore the model's capability for autonomous rollout and its potential to generate ``what-if" scenarios. Extensive experiments show that GEM achieves state-of-the-art performances across diverse benchmarks and evaluation settings, demonstrating its superiority and effectiveness. Project page: https://github.com/wuyang98/GEM.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
49 of 61 references.
Closest ancestors · 11
Same direction, newest first
- 2025Learning to Generate 4D LiDAR Sequences15 Sep
- 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences5 Aug
- 2025WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusion18 Apr
- 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformer29 Jul
- 2024Towards Realistic Scene Generation with LiDAR Diffusion Models31 Mar
- 2023LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models17 Sep
- 2022Learning to Generate Realistic LiDAR Point Clouds8 Sep
- 2021Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks28 Sep
- 2021Learning to Drop Points for LiDAR Scan Synthesis23 Feb
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space7 Jun
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation2 Dec
Other references · 38
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
- 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
- 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
- 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
- 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
26 more
- 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
- 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
- 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
- 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
- 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
- 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
- 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
- 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
- 2022PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detectioncited by 99
- 2021Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weathercited by 95
- 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
- 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecastingcited by 76
- 2023UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generationcited by 76
- 2022LiDAR Snowfall Simulation for Robust 3D Object Detectioncited by 72
- 20253D and 4D World Modeling: A Surveycited by 64
- 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
- 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
- 2021Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detectioncited by 42
- 2021To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernelscited by 39
- 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
- 2025DefMamba: Deformable Visual State Space Modelcited by 18
- 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
- 2025Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusioncited by 11
- 2025Learning Class Prototypes for Unified Sparse Supervised 3D Object Detectioncited by 10
- 2025A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generationcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026LiAuto-GeoX: Efficient Grounded Driving Transformercited by 0
- 2026LiAuto-GeoX: Efficient Grounded Driving Transformer4 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2024LidarDM: Generative LiDAR Simulation in a Generated World
- 2025LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Driving
- 2025LaGen: Towards Autoregressive LiDAR Scene Generation
- 2025U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences
- 2024LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenes
- 2026WOLF: World Model Guided LiDAR Exploration with Predictive Frontiers
- 2025Towards foundational LiDAR world models with efficient latent flow matching
- 2025DriveLiDAR4D: Sequential and Controllable LiDAR Scene Generation for Autonomous Driving