mlsanityData to 7 Oct 2026

Geometry-Free View Synthesis: Transformers and no 3D Priors

arXiv 2104.07652 ↗First posted 15 Apr 2021cs.CVcited by 99

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Is a geometric model required to synthesize novel views from a single image? Being bound to local convolutions, CNNs need explicit 3D biases to model geometric transformations. In contrast, we demonstrate that a transformer-based model can synthesize entirely novel views without any hand-engineered 3D biases. This is achieved by (i) a global attention mechanism for implicitly learning long-range 3D correspondences between source and target views, and (ii) a probabilistic formulation necessary to capture the ambiguity inherent in predicting novel views from a single image, thereby overcoming the limitations of previous approaches that are restricted to relatively small viewpoint changes. We evaluate various ways to integrate 3D priors into a transformer architecture. However, our experiments show that no such geometric priors are required and that the transformer is capable of implicitly learning 3D relationships between images. Furthermore, this approach outperforms the state of the art in terms of visual quality while covering the full distribution of possible realizations. Code is available at https://git.io/JOnwn

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 51 of its 84 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

51 of 84 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2016Layer Normalizationcited by 5,579
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
38 more
  1. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  2. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  3. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  4. 2020Jukebox: A Generative Model for Musiccited by 535
  5. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  6. 2016Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescuecited by 368
  7. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  8. 2019Neural Point-Based Graphicscited by 326
  9. 2015A note on the evaluation of generative modelscited by 291
  10. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  11. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  12. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  13. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  14. 2020Free View Synthesiscited by 231
  15. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  16. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  17. 2019Diagnosing and Enhancing VAE Modelscited by 192
  18. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  19. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  20. 2016Variational Lossy Autoencodercited by 175
  21. 2016View Synthesis by Appearance Flowcited by 172
  22. 2020Stable View Synthesiscited by 165
  23. 2016Learning-Based View Synthesis for Light Field Camerascited by 145
  24. 20193D Ken Burns Effect from a Single Imagecited by 128
  25. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  26. 2019Neural Rerendering in the Wildcited by 121
  27. 2018Extreme View Synthesiscited by 94
  28. 2020Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Imagecited by 77
  29. 2016Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networkscited by 72
  30. 2020Crowdsampling the Plenoptic Functioncited by 62
  31. 2017Learning to Synthesize a 4D RGBD Light Field from a Single Imagecited by 60
  32. 2017Transformation-Grounded Image Generation Network for Novel 3D View Synthesiscited by 59
  33. 2015Multi-view 3D Models from Single Images with a Convolutional Networkcited by 57
  34. 2020A Disentangling Invertible Interpretation Network for Explaining Latent Representationscited by 57
  35. 2020One Shot 3D Photographycited by 45
  36. 2014Learning to Generate Chairs, Tables and Cars with Convolutional Networkscited by 26
  37. 2016Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesiscited by 24
  38. 2018Peeking Behind Objects: Layered Depth Prediction from a Single Imagecited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 23; 2023: 25; 2024: 21; 2025: 19; 2026: 8.

99 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  3. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  4. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  5. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  6. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  7. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  8. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  9. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  10. 2022Generalizable Patch-Based Neural Renderingcited by 121
  11. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  12. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
38 more
  1. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  2. 2021Light Field Neural Renderingcited by 108
  3. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  4. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  5. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 91
  6. 2022Object Scene Representation Transformercited by 90
  7. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesiscited by 84
  8. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  9. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  10. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  11. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  12. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  13. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 72
  14. 2022ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformerscited by 70
  15. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  16. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  17. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  18. 2024Advances in 3D Generation: A Surveycited by 62
  19. 2022Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Imagescited by 59
  20. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  21. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  22. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  23. 2022Implicit Warping for Animation with Image Setscited by 45
  24. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Modelscited by 45
  25. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  26. 2022FWD: Real-time Novel View Synthesis with Forward Warping and Depthcited by 39
  27. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  28. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  29. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  30. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Viewscited by 31
  31. 2024G3R: Gradient Guided Generalizable Reconstructioncited by 29
  32. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  33. 2023LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Modelcited by 25
  34. 20253D Scene Generation: A Surveycited by 24
  35. 2022Aerial Monocular 3D Object Detectioncited by 23
  36. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  37. 2022Towards Practical Plug-and-Play Diffusion Modelscited by 20
  38. 2025Positional Encoding Fieldcited by 20

50 most cited of 99 All 99 (JSON)