mlsanityData to 7 Oct 2026

Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

arXiv 2110.04994 ↗First posted 11 Oct 2021cs.CV cs.AI cs.GRcited by 336

Not on the map. Why?

Cited by 61 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. In addition to enabling interesting lines of research, we show the tooling and generated data suffice to train robust vision models. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks, despite having seen no benchmark or non-pipeline data. The depth estimation network outperforms MiDaS and the surface normal estimation network is the first to achieve human-level performance for in-the-wild surface normal estimation -- at least according to one metric on the OASIS benchmark. The Dockerized pipeline with CLI, the (mostly python) code, PyTorch dataloaders for the generated data, the generated starter dataset, download scripts and other utilities are available through our project website, https://omnidata.vision.

Key facts. Cited by 336 papers in the map (data through 7 Oct 2026). Builds on 48 of its 74 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

48 of 74 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 2016Matching Networks for One Shot Learningcited by 2,246
36 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  10. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  11. 2019Panoptic Feature Pyramid Networkscited by 644
  12. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  13. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  14. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  15. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  16. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  17. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  18. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  19. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  20. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  21. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  22. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  23. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  24. 2016Single-Image Depth Perception in the Wildcited by 179
  25. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  26. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  27. 2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
  28. 2020Robust Learning Through Cross-Task Consistencycited by 105
  29. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  30. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  31. 2020OASIS: A Large-Scale Dataset for Single Image 3D in the Wildcited by 62
  32. 2019The Omniglot challenge: a 3-year progress reportcited by 54
  33. 2016Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networkscited by 35
  34. 2017MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimationcited by 21
  35. 2018Left-Right Comparative Recurrent Model for Stereo Matchingcited by 12
  36. 2017Surface Normals in the Wildcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 20; 2023: 76; 2024: 118; 2025: 76; 2026: 45.

336 papers.

Sort:

By area:

  1. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  2. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  3. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 375
  4. 2022Kubric: A scalable dataset generatorcited by 373
  5. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  6. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 357
  7. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  8. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  9. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  10. 2024MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervisioncited by 290
  11. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  12. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
38 more
  1. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 275
  2. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  3. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  4. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  5. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  6. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  7. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  8. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
  9. 2023NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAMcited by 162
  10. 2024High-quality Surface Reconstruction using Gaussian Surfelscited by 159
  11. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  12. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  13. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  14. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 141
  15. 2024DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshingcited by 136
  16. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  17. 20223D Common Corruptions and Data Augmentationcited by 120
  18. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  19. 20234M: Massively Multimodal Masked Modelingcited by 118
  20. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 113
  21. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 100
  22. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  23. 2025Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridgingcited by 95
  24. 2024MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstructioncited by 94
  25. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  26. 2024Fine-Tuning Image-Conditional Diffusion Models is Easier than You Thinkcited by 82
  27. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  28. 2024DepthFM: Fast Monocular Depth Estimation with Flow Matchingcited by 76
  29. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  30. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  31. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimationcited by 73
  32. 2024How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Surveycited by 72
  33. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  34. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  35. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  36. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priorscited by 64
  37. 2024Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussianscited by 64
  38. 2025WorldMirror: Universal 3D World Reconstruction with Any-Prior Promptingcited by 60

50 most cited of 336 All 336 (JSON)