mlsanityData to 7 Oct 2026

Recognizing Scenes from Novel Viewpoints

arXiv 2112.01520 ↗First posted 2 Dec 2021cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Humans can perceive scenes in 3D from a handful of 2D views. For AI agents, the ability to recognize a scene from any viewpoint given only a few images enables them to efficiently interact with the scene and its objects. In this work, we attempt to endow machines with this ability. We propose a model which takes as input a few RGB images of a new scene and recognizes the scene from novel viewpoints by segmenting it into semantic categories. All this without access to the RGB images from those views. We pair 2D scene recognition with an implicit 3D representation and learn from multi-view 2D annotations of hundreds of scenes without any 3D supervision beyond camera poses. We experiment on challenging datasets and demonstrate our model's ability to jointly capture semantics and geometry of novel scenes with diverse layouts, object types and shapes.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

47 of 74 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  12. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
35 more
  1. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  2. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  3. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  4. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  5. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  6. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  7. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  8. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  9. 2017Neural 3D Mesh Renderercited by 437
  10. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  11. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  12. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  13. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  14. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  15. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  16. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  17. 2020Consistent Video Depth Estimationcited by 220
  18. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  19. 2017Learning a Multi-View Stereo Machinecited by 194
  20. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  21. 2016Single-Image Depth Perception in the Wildcited by 179
  22. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  23. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  24. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  25. 2020Articulation-aware Canonical Surface Mappingcited by 80
  26. 2021Unsupervised Learning of 3D Object Categories from Videos in the Wildcited by 67
  27. 2021Shelf-Supervised Mesh Prediction in the Wildcited by 65
  28. 2017Learning to Synthesize a 4D RGBD Light Field from a Single Imagecited by 60
  29. 2017Predicting Deeper into the Future of Semantic Segmentationcited by 51
  30. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  31. 2014Convolutional Feature Masking for Joint Object and Stuff Segmentationcited by 41
  32. 2021Planar Surface Reconstruction from Sparse Viewscited by 40
  33. 2019Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstructioncited by 28
  34. 2018Learning Single-Image Depth from Videos using Quality Assessment Networkscited by 25
  35. 2020Associative3D: Volumetric Reconstruction from Sparse Viewscited by 18

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  2. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  3. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  4. 2023SSR-2D: Semantic 3D Scene Reconstruction from 2D Imagescited by 7
  5. 2023Learning to Predict Scene-Level Implicit 3D from Posed RGBD Datacited by 2