# DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes

arXiv: https://arxiv.org/abs/2412.19458 · HTML: https://mlsanity.com/paper/2412.19458/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)
- [MagicDrive: Street View Generation with Diverse 3D Geometry Control](https://mlsanity.com/paper/2310.02601/) (2023)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout](https://mlsanity.com/paper/2308.01661/) (2023)
- [Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents](https://mlsanity.com/paper/2402.05746/) (2024)
- [DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model](https://mlsanity.com/paper/2310.07771/) (2023)
- [Edit-A-Video: Single Video Editing with Object-Aware Consistency](https://mlsanity.com/paper/2303.07945/) (2023)
- [Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation](https://mlsanity.com/paper/2406.01349/) (2024)
- [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024)
- [VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing](https://mlsanity.com/paper/2306.08707/) (2023)
- [SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control](https://mlsanity.com/paper/2403.19438/) (2024)
- [InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing](https://mlsanity.com/paper/2308.00135/) (2023)
- [EVE: Efficient zero-shot text-based Video Editing with Depth Map Guidance and Temporal Consistency Constraints](https://mlsanity.com/paper/2308.10648/) (2023)
- [RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing](https://mlsanity.com/paper/2312.12635/) (2023)

## Built on by (most cited first, 17 of 17)

- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [SceneCrafter: Controllable Multi-View Driving Scene Editing](https://mlsanity.com/paper/2506.19488/) (2025)
- [Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation](https://mlsanity.com/paper/2508.20471/) (2025)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [GS-RoadPatching: Inpainting Gaussians via 3D Searching and Placing for Driving Scenes](https://mlsanity.com/paper/2509.19937/) (2025)
- [LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents](https://mlsanity.com/paper/2512.17445/) (2025)
- [Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenes](https://mlsanity.com/paper/2512.24227/) (2025)
- [HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles](https://mlsanity.com/paper/2602.21333/) (2026)
- [Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation](https://mlsanity.com/paper/2603.12864/) (2026)
- [HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes](https://mlsanity.com/paper/2604.04887/) (2026)
- [HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2511.22187/) (2025)
- [DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation](https://mlsanity.com/paper/2602.22549/) (2026)
- [VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning](https://mlsanity.com/paper/2603.28353/) (2026)
- [FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model](https://mlsanity.com/paper/2606.20110/) (2026)
- [MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priors](https://mlsanity.com/paper/2507.21872/) (2025)
- [SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis](https://mlsanity.com/paper/2512.22706/) (2025)
- [DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation](https://mlsanity.com/paper/2606.31918/) (2026)

## Abstract (arXiv)

Vision-centric autonomous driving systems require diverse data for robust training and evaluation, which can be augmented by manipulating object positions and appearances within existing scene captures. While recent advancements in diffusion models have shown promise in video editing, their application to object manipulation in driving scenarios remains challenging due to imprecise positional control and difficulties in preserving high-fidelity object appearances. To address these challenges in position and appearance control, we introduce DriveEditor, a diffusion-based framework for object editing in driving videos. DriveEditor offers a unified framework for comprehensive object editing operations, including repositioning, replacement, deletion, and insertion. These diverse manipulations are all achieved through a shared set of varying inputs, processed by identical position control and appearance maintenance modules. The position control module projects the given 3D bounding box while preserving depth information and hierarchically injects it into the diffusion process, enabling precise control over object position and orientation. The appearance maintenance module preserves consistent attributes with a single reference image by employing a three-tiered approach: low-level detail preservation, high-level semantic maintenance, and the integration of 3D priors from a novel view synthesis model. Extensive qualitative and quantitative evaluations on the nuScenes dataset demonstrate DriveEditor's exceptional fidelity and controllability in generating diverse driving scene edits, as well as its remarkable ability to facilitate downstream tasks. Project page: https://yvanliang.github.io/DriveEditor.