# USR-Drive: Unified Driving Scene Representation via Joint Denoising of 3D Gaussians and Boxes

arXiv: https://arxiv.org/abs/2608.19036 · HTML: https://mlsanity.com/paper/2608.19036/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 30 of 32)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [Shape of Motion: 4D Reconstruction from a Single Video](https://mlsanity.com/paper/2407.13764/) (2024)
- [AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views](https://mlsanity.com/paper/2505.23716/) (2025)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [End-to-End Driving with Online Trajectory Evaluation via BEV World Model](https://mlsanity.com/paper/2504.01941/) (2025)
- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control](https://mlsanity.com/paper/2411.13807/) (2024)
- [MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries](https://mlsanity.com/paper/2205.00613/) (2022)
- [MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes](https://mlsanity.com/paper/2405.14475/) (2024)
- [GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction](https://mlsanity.com/paper/2412.10373/) (2024)
- [STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes](https://mlsanity.com/paper/2501.00602/) (2024)
- [Efficiently Reconstructing Dynamic Scenes One D4RT at a Time](https://mlsanity.com/paper/2512.08924/) (2025)
- [Temporal Enhanced Training of Multi-view 3D Object Detector via Historical Object Prediction](https://mlsanity.com/paper/2304.00967/) (2023)
- [DriveLaW:Unifying Planning and Video Generation in a Latent Driving World](https://mlsanity.com/paper/2512.23421/) (2025)
- [ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation](https://mlsanity.com/paper/2510.23306/) (2025)
- [Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction](https://mlsanity.com/paper/2601.04090/) (2026)
- [DGGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images](https://mlsanity.com/paper/2512.03004/) (2025)
- [4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere](https://mlsanity.com/paper/2602.10094/) (2026)
- [Repurposing Geometric Foundation Models for Multi-view Diffusion](https://mlsanity.com/paper/2603.22275/) (2026)

## Abstract (arXiv)

Spatial representation learning for autonomous driving aims to map raw visual signals into structured 3D scene representations, where object-centric bounding boxes and rendering-oriented 3D primitives (eg, 3D Gaussians) serve as two distinct yet highly complementary levels for scene understanding. Existing methods typically treat dynamic reconstruction and instance-level perception as separate tasks, despite their shared goal of estimating the underlying 3D world state. As a result, dynamic reconstruction is under-constrained while 3D detection lacks geometric grounding. To address this gap, we propose USR-Drive, a unified conditional generative framework that, given only posed multi-view driving videos, jointly recovers dense dynamic geometry and instance-level object layouts within a shared scene representation. Specifically, USR-Drive represents dense Gaussian primitives and sparse 3D bounding boxes as two aligned latent token streams and jointly denoises them with a unified multi-modal diffusion Transformer. Unlike prior paradigms that use boxes as external conditions or predict them with detached modules, USR-Drive treats them as mutually constrained state variables with a Unified Positional Encoding (UPE) that aligns heterogeneous tokens within a shared metric spatiotemporal coordinate. Via such unified representation and generative framework, the two modalities reinforce each other: geometry supplies dense metric evidence for box prediction, while boxes provide instance-level structural priors that help preserve spatial consistency and reduce ambiguity in sequential 3D geometric representation. Our approach successfully delivers state-of-the-art results for both dynamic reconstruction and 3D detection on the nuScenes and VKitti datasets.