mlsanityData to 7 Oct 2026

Panacea: Panoramic and Controllable Video Generation for Autonomous Driving

arXiv 2311.16813 ↗First posted 28 Nov 2023cs.CVcited by 165

Not on the map. Why?

Cited by 63 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The field of autonomous driving increasingly demands high-quality annotated training data. In this paper, we propose Panacea, an innovative approach to generate panoramic and controllable videos in driving scenarios, capable of yielding an unlimited numbers of diverse, annotated samples pivotal for autonomous driving advancements. Panacea addresses two critical challenges: 'Consistency' and 'Controllability.' Consistency ensures temporal and cross-view coherence, while Controllability ensures the alignment of generated content with corresponding annotations. Our approach integrates a novel 4D attention and a two-stage generation pipeline to maintain coherence, supplemented by the ControlNet framework for meticulous control by the Bird's-Eye-View (BEV) layouts. Extensive qualitative and quantitative evaluations of Panacea on the nuScenes dataset prove its effectiveness in generating high-quality multi-view driving-scene videos. This work notably propels the field of autonomous driving by effectively augmenting the training dataset used for advanced BEV perception techniques.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Builds on 58 of its 62 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

58 of 62 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  13. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  14. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  15. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  16. 2023ModelScope Text-to-Video Technical Reportcited by 651
  17. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  18. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  19. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  20. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  21. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  22. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  23. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 311
  24. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  25. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  26. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  27. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
  28. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  29. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  30. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  31. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  32. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  33. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  34. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  35. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  36. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  37. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  38. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 31; 2025: 80; 2026: 53.

165 papers.

Sort:

By area:

  1. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  4. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  5. 2025Controllable Video Generation: A Surveycited by 93
  6. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  7. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  8. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  9. 20253D and 4D World Modeling: A Surveycited by 64
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  12. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
38 more
  1. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  2. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  3. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  4. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  5. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 35
  6. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  7. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 34
  8. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  9. 2024SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Controlcited by 32
  10. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  11. 2024Vision-based 3D occupancy prediction in autonomous driving: a review and outlookcited by 29
  12. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  13. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27
  14. 20253D Scene Generation: A Surveycited by 24
  15. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  16. 2025A Survey of Interactive Generative Videocited by 23
  17. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  18. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
  19. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  20. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  21. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  22. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  23. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  24. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  25. 2024SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIscited by 14
  26. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  27. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  28. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
  29. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  30. 2024X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarioscited by 13
  31. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13
  32. 2024PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Modelscited by 12
  33. 2024Physical Informed Driving World Modelcited by 12
  34. 2025Omnidirectional Multi-Object Trackingcited by 12
  35. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  36. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  37. 2025Decoupled Diffusion Sparks Adaptive Scene Generationcited by 12
  38. 2024DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Drivingcited by 11

50 most cited of 165 All 165 (JSON)