mlsanityData to 7 Oct 2026

Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformer

arXiv 2407.19628 ↗First posted 29 Jul 2024cs.CVcited by 26

On the map: World models → World modeling from LiDAR generation

Abstract (arXiv)

The complex traffic environment and various weather conditions make the collection of LiDAR data expensive and challenging. Achieving high-quality and controllable LiDAR data generation is urgently needed, controlling with text is a common practice, but there is little research in this field. To this end, we propose Text2LiDAR, the first efficient, diverse, and text-controllable LiDAR data generation model. Specifically, we design an equirectangular transformer architecture, utilizing the designed equirectangular attention to capture LiDAR features in a manner with data characteristics. Then, we design a control-signal embedding injector to efficiently integrate control signals through the global-to-focused attention mechanism. Additionally, we devise a frequency modulator to assist the model in recovering high-frequency details, ensuring the clarity of the generated point cloud. To foster development in the field and optimize text-controlled generation performance, we construct nuLiDARtext which offers diverse text descriptors for 34,149 LiDAR point clouds from 850 scenes. Experiments on uncontrolled and text-controlled generation in various forms on KITTI-360 and nuScenes datasets demonstrate the superiority of our approach.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 70 of its 86 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

70 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models17 Sep
  2. 2022Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data21 Oct
  3. 2022Learning to Generate Realistic LiDAR Point Clouds8 Sep
  4. 2021Learning to Drop Points for LiDAR Scan Synthesis23 Feb
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation2 Dec

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
38 more
  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  10. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  11. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  12. 2023ModelScope Text-to-Video Technical Reportcited by 651
  13. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  14. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  15. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  16. 2023Text-to-3D using Gaussian Splattingcited by 299
  17. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  18. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  19. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  20. 2021Projected GANs Converge Fastercited by 204
  21. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  22. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  23. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  24. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  25. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  26. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  27. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  28. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 128
  29. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  30. 2022Diffusion Probabilistic Model Made Slimcited by 121
  31. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  32. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  33. 2020Discrete Point Flow Networks for Efficient Point Cloud Generationcited by 87
  34. 2021RigNet: Repetitive Image Guided Network for Depth Completioncited by 80
  35. 2018Deep Generative Modeling of LiDAR Datacited by 70
  36. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  37. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  38. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 6.

26 papers.

Sort:

By area:

  1. 20253D and 4D World Modeling: A Surveycited by 64
  2. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Modelscited by 22
  3. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  4. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 14
  5. 2025WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusioncited by 12
  6. 2025SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understandingcited by 10
  7. 2025La La LiDAR: Large-Scale Layout Generation from LiDAR Datacited by 10
  8. 2024LOGen: Toward Lidar Object Generation by Point Diffusioncited by 9
  9. 2025Controllable 3D Outdoor Scene Generation via Scene Graphscited by 9
  10. 2025Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scenecited by 5
  11. 2025Veila: Panoramic LiDAR Generation from a Monocular RGB Imagecited by 5
  12. 2025U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequencescited by 5
14 more
  1. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  2. 2025A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generationcited by 4
  3. 2025SG-LDM: Semantic-Guided LiDAR Generation via Latent-Aligned Diffusioncited by 2
  4. 2025Learning to Generate 4D LiDAR Sequencescited by 2
  5. 2025RadarGen: Automotive Radar Point Cloud Generation from Camerascited by 2
  6. 2026Few-Shot Incremental 3D Object Detection in Dynamic Indoor Environmentscited by 2
  7. 2025DriveLiDAR4D: Sequential and Controllable LiDAR Scene Generation for Autonomous Drivingcited by 1
  8. 2025LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputscited by 1
  9. 2026GEM: Generating LiDAR World Model via Deformable Mambacited by 1
  10. 2025MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priorscited by 0
  11. 2026OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generationcited by 0
  12. 2026T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generationcited by 0
  13. 2026SUMI: Scalable Unified Model for 3D Point Cloud Inferencecited by 0
  14. 2026Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Range-Edit: Semantic Mask Guided Outdoor LiDAR Scene Editing
  2. 2022LidarCLIP or: How I Learned to Talk to Point Clouds
  3. 2024RangeLDM: Fast Realistic LiDAR Point Cloud Generation
  4. 2025TopoLiDM: Topology-Aware LiDAR Diffusion Models for Interpretable and Realistic LiDAR Point Cloud Generation
  5. 2024LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenes
  6. 2025PVNet: Point-Voxel Interaction LiDAR Scene Upsampling Via Diffusion Models
  7. 2025B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding
  8. 2025LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Driving