mlsanityData to 14 Oct 2026

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

arXiv 2109.13410 ↗First posted 28 Sep 2021cs.CVcited by 575

Not on the map. Why?

Cited by 39 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

Key facts. Cited by 575 papers in the map (data through 14 Oct 2026). Builds on 60 of its 122 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

60 of 122 references.

References · 60

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2017Mask R-CNNcited by 7,277
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2016Pyramid Scene Parsing Networkcited by 2,479
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  12. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
38 more
  1. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  3. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  4. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 806
  5. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  6. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  7. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2018Learning to Adapt Structured Output Space for Semantic Segmentationcited by 584
  10. 2020Virtual KITTI 2cited by 353
  11. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  12. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  13. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  14. 2016FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptationcited by 243
  15. 2020Free View Synthesiscited by 231
  16. 2018The ApolloScape Open Dataset for Autonomous Driving and its Applicationcited by 190
  17. 2020A2D2: Audi Autonomous Driving Datasetcited by 190
  18. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 190
  19. 2016Deep Feature Flow for Video Recognitioncited by 186
  20. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 184
  21. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  22. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  23. 2017Semantic3D.net: A new Large-scale Point Cloud Classification Benchmarkcited by 158
  24. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
  25. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 144
  26. 2020Video Panoptic Segmentationcited by 142
  27. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  28. 2018Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++cited by 120
  29. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  30. 2018CubeSLAM: Monocular 3D Object SLAMcited by 102
  31. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  32. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 96
  33. 2019Fast Interactive Object Annotation with Curve-GCNcited by 95
  34. 2017Annotating Object Instances with a Polygon-RNNcited by 88
  35. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  36. 2020Toronto-3D: A Large-scale Mobile LiDAR Dataset for Semantic Segmentation of Urban Roadwayscited by 77
  37. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  38. 2019Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priorscited by 74

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 59; 2023: 105; 2024: 141; 2025: 160; 2026: 105.

575 papers.

Sort:

By area:

  1. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  2. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  3. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  4. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  5. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  6. 2021Urban Radiance Fieldscited by 263
  7. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  8. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  9. 2021DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motioncited by 217
  10. 2021Projected GANs Converge Fastercited by 208
  11. 2023MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyondcited by 185
  12. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 158
38 more
  1. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 157
  2. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  3. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  4. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  5. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  6. 2023SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenescited by 131
  7. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  8. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  9. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
  10. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  11. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  12. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 93
  13. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 90
  14. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 90
  15. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 86
  16. 2023UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generationcited by 78
  17. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 75
  18. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 72
  19. 2024Towards Realistic Scene Generation with LiDAR Diffusion Modelscited by 69
  20. 20253D and 4D World Modeling: A Surveycited by 67
  21. 20223D Object Detection from Images for Autonomous Driving: A Surveycited by 66
  22. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
  23. 2023Behind the Scenes: Density Fields for Single View Reconstructioncited by 63
  24. 2024OmniSat: Self-Supervised Modality Fusion for Earth Observationcited by 63
  25. 2025UniK3D: Universal Camera Monocular 3D Estimationcited by 62
  26. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  27. 2025A Survey of World Models for Autonomous Drivingcited by 60
  28. 2022Waymo Open Dataset: Panoramic Video Panoptic Segmentationcited by 58
  29. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  30. 2023Efficient 3D Semantic Segmentation with Superpoint Transformercited by 57
  31. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  32. 2023LiDAR Data Synthesis with Denoising Diffusion Probabilistic Modelscited by 55
  33. 2022Uncertainty-aware Vision-based Metric Cross-view Geolocalizationcited by 54
  34. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 54
  35. 2023NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenescited by 53
  36. 2024Context and Geometry Aware Voxel Transformer for Semantic Scene Completioncited by 50
  37. 2024Point-SAM: Promptable 3D Segmentation Model for Point Cloudscited by 50
  38. 2022Vox-Surf: Voxel-based Implicit Surface Representationcited by 49

50 most cited of 575 All 575 (JSON)