mlsanityData to 14 Oct 2026

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

arXiv 2011.02523 ↗First posted 4 Nov 2020cs.CV cs.GRcited by 604

Not on the map. Why?

Cited by 88 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations and complete camera information for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, computation time, and annotation effort. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a popular open-source natural language processing model. We also evaluate sim-to-real transfer performance on two real-world scene understanding tasks - semantic segmentation and 3D shape prediction - where we find that pre-training on our dataset significantly improves performance on both tasks, and achieves state-of-the-art performance on the most challenging Pix3D test set. All of our rendered image data, as well as all the code we used to generate our dataset and perform our experiments, is available online.

Key facts. Cited by 604 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 39; 2023: 54; 2024: 115; 2025: 165; 2026: 215.

604 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,234
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  5. 2024Depth Anything V2cited by 1,422
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  7. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
  8. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  9. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  10. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
  11. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  12. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
38 more
  1. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  2. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  3. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  6. 2024MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervisioncited by 299
  7. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  8. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
  9. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 261
  10. 2025AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Viewscited by 243
  11. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 174
  12. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 167
  13. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 166
  14. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion modelscited by 161
  15. 2025Streaming 4D Visual Geometry Transformercited by 161
  16. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  17. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 156
  18. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  19. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 154
  20. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  21. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  22. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 139
  23. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
  24. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 137
  25. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  26. 2023DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Modelscited by 127
  27. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 124
  28. 20234M: Massively Multimodal Masked Modelingcited by 120
  29. 20223D Common Corruptions and Data Augmentationcited by 119
  30. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  31. 2025Visual Spatial Tuningcited by 89
  32. 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 87
  33. 2024Fine-Tuning Image-Conditional Diffusion Models is Easier than You Thinkcited by 83
  34. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 82
  35. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  36. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 77
  37. 2024DepthFM: Fast Monocular Depth Estimation with Flow Matchingcited by 76
  38. 2022Offline Visual Representation Learning for Embodied Navigationcited by 75

50 most cited of 604 All 604 (JSON)