mlsanityData to 7 Oct 2026

Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

arXiv 2111.13152 ↗First posted 25 Nov 2021cs.CV cs.AI cs.GRcited by 222

Not on the map. Why?

Cited by 37 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

36 of 54 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  9. 2021Vision Transformers for Dense Predictioncited by 1,789
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  11. 2020Point Transformercited by 1,462
  12. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
24 more
  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  3. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  4. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  5. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  6. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  7. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  8. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  9. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  10. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  11. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  12. 2017Neural 3D Mesh Renderercited by 437
  13. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  14. 2021Advances in Neural Renderingcited by 365
  15. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  16. 2020State of the Art on Neural Renderingcited by 276
  17. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  18. 2016EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesiscited by 177
  19. 2019Neural Rerendering in the Wildcited by 121
  20. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 111
  21. 2018Learning to Navigate in Cities Without a Mapcited by 103
  22. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  23. 2017Mesh-based 3D Textured Urban Mappingcited by 5
  24. 2016Depth Estimation Through a Generative Model of Light Field Synthesiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 25; 2023: 71; 2024: 47; 2025: 44; 2026: 34.

222 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  3. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  4. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  5. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
  6. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  7. 2022Kubric: A scalable dataset generatorcited by 373
  8. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  9. 2022Novel View Synthesis with Diffusion Modelscited by 299
  10. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  11. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  12. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
38 more
  1. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  2. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 176
  3. 2023PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Predictioncited by 165
  4. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  5. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  6. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  7. 2022Generalizable Patch-Based Neural Renderingcited by 121
  8. 2022Is Attention All That NeRF Needs?cited by 113
  9. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  10. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  11. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  12. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 111
  13. 2021Light Field Neural Renderingcited by 108
  14. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  15. 2023Learning to Render Novel Views from Wide-Baseline Stereo Pairscited by 103
  16. 2025Cameras as Relative Positional Encodingcited by 94
  17. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  18. 2022Object Scene Representation Transformercited by 90
  19. 2023SHERF: Generalizable Human NeRF from a Single Imagecited by 88
  20. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  21. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesiscited by 84
  22. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  23. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  24. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 76
  25. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
  26. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  27. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
  28. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  29. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  30. 2023MuRF: Multi-Baseline Radiance Fieldscited by 70
  31. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  32. 2023LEAP: Liberate Sparse-view 3D Modeling from Camera Posescited by 65
  33. 2024Controlling Space and Time with Diffusion Modelscited by 65
  34. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  35. 2023Behind the Scenes: Density Fields for Single View Reconstructioncited by 63
  36. 2024Advances in 3D Generation: A Surveycited by 62
  37. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  38. 2022Improving Object-centric Learning with Query Optimizationcited by 60

50 most cited of 222 All 222 (JSON)