mlsanityData to 14 Oct 2026

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

arXiv 2203.10157 ↗First posted 18 Mar 2022cs.CV cs.LGcited by 70

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Novel view synthesis is a long-standing problem. In this work, we consider a variant of the problem where we are given only a few context views sparsely covering a scene or an object. The goal is to predict novel viewpoints in the scene, which requires learning priors. The current state of the art is based on Neural Radiance Field (NeRF), and while achieving impressive results, the methods suffer from long training times as they require evaluating millions of 3D point samples via a neural network for each image. We propose a 2D-only method that maps multiple context views and a query pose to a new image in a single pass of a neural network. Our model uses a two-stage architecture consisting of a codebook and a transformer model. The codebook is used to embed individual images into a smaller latent space, and the transformer solves the view synthesis task in this more compact space. To train our model efficiently, we introduce a novel branching attention mechanism that allows us to use the same model not only for neural rendering but also for camera pose estimation. Experimental results on real-world scenes show that our approach is competitive compared to NeRF-based methods while not reasoning explicitly in 3D, and it is faster to train.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 55 of its 83 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

55 of 83 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
38 more
  1. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  3. 2018Group Normalizationcited by 1,524
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  6. 2020Neural Sparse Voxel Fieldscited by 959
  7. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  10. 2018Image Transformercited by 772
  11. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  12. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  13. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  14. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 559
  15. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 511
  16. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  17. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
  18. 2020Free View Synthesiscited by 231
  19. 2017Geometric Loss Functions for Camera Pose Regression with Deep Learningcited by 224
  20. 2021Back to the Feature: Learning Robust Camera Localization from Pixels to Posecited by 199
  21. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  22. 2019Understanding the Limitations of CNN-based Absolute Camera Pose Regressioncited by 177
  23. 2020Visual Camera Re-Localization from RGB and RGB-D Images Using DSACcited by 176
  24. 2020Stable View Synthesiscited by 165
  25. 2017Geometry-Aware Learning of Maps for Camera Localizationcited by 161
  26. 2015Modelling Uncertainty in Deep Learning for Camera Relocalizationcited by 134
  27. 2021LENS: Localization enhanced by NeRF synthesiscited by 117
  28. 2021Learning Multi-Scene Absolute Pose Regression with Transformerscited by 108
  29. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 100
  30. 2018Extreme View Synthesiscited by 94
  31. 2018InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Datasetcited by 87
  32. 2017Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Networkcited by 79
  33. 2021Unsupervised Learning of 3D Object Categories from Videos in the Wildcited by 67
  34. 2019To Learn or Not to Learn: Visual Localization from Essential Matricescited by 61
  35. 2021Direct-PoseNet: Absolute Pose Regression with Photometric Consistencycited by 61
  36. 2017Relative Camera Pose Estimation Using Convolutional Neural Networkscited by 47
  37. 2019Large-scale, real-time visual-inertial localization revisitedcited by 45
  38. 2018Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascadecited by 35

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 32; 2024: 19; 2025: 11; 2026: 4.

70 papers.

Sort:

By area:

  1. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  2. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 364
  3. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  4. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  5. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 274
  6. 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
  7. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  8. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 189
  9. 2024CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularizationcited by 121
  10. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Imagescited by 106
  11. 2025Cameras as Relative Positional Encodingcited by 105
  12. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 89
38 more
  1. 2023PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstructioncited by 84
  2. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesiscited by 84
  3. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 79
  4. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  5. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 68
  6. 2024Advances in 3D Generation: A Surveycited by 63
  7. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58
  8. 2023Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedracited by 57
  9. 2024LaRa: Efficient Large-Baseline Radiance Fieldscited by 55
  10. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  11. 2024VideoMV: Consistent Multi-View Generation Based on Large Video Generative Modelcited by 45
  12. 2023FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flowcited by 39
  13. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  14. 2023Novel-View Acoustic Synthesiscited by 33
  15. 2023LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFscited by 33
  16. 2023A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstructioncited by 32
  17. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Viewscited by 31
  18. 2024G3R: Gradient Guided Generalizable Reconstructioncited by 31
  19. 2022Common Pets in 3D: Dynamic New-View Synthesis of Real-Life Deformable Categoriescited by 29
  20. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Expertscited by 27
  21. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  22. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 27
  23. 2024NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methodscited by 26
  24. 2023Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Viewscited by 25
  25. 2023Improving NeRF Quality by Progressive Camera Placement for Unrestricted Navigation in Complex Environmentscited by 19
  26. 2023NViST: In the Wild New View Synthesis from a Single Image with Transformerscited by 19
  27. 2024Consolidating Attention Features for Multi-view Image Editingcited by 19
  28. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  29. 2024GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstructioncited by 18
  30. 2024Denoising Diffusion via Image-Based Renderingcited by 15
  31. 2022A Neural ODE Interpretation of Transformer Layerscited by 14
  32. 2023UpFusion: Novel View Diffusion from Unposed Sparse View Observationscited by 14
  33. 2024Lightplane: Highly-Scalable Components for Neural 3D Fieldscited by 13
  34. 2023Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputscited by 11
  35. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 11
  36. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Modelscited by 10
  37. 2025CoMapGS: Covisibility Map-based Gaussian Splatting for Sparse Novel View Synthesiscited by 10
  38. 2023Geometry-biased Transformers for Novel View Synthesiscited by 9

50 most cited of 70 All 70 (JSON)