# NViST: In the Wild New View Synthesis from a Single Image with Transformers

arXiv: https://arxiv.org/abs/2312.08568 · HTML: https://mlsanity.com/paper/2312.08568/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 101 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 91)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)

## Built on by (most cited first, 19 of 19)

- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting](https://mlsanity.com/paper/2402.10259/) (2024)
- [PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence](https://mlsanity.com/paper/2411.16877/) (2024)
- [TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models](https://mlsanity.com/paper/2411.18350/) (2024)
- [Novel View Synthesis with Pixel-Space Diffusion Models](https://mlsanity.com/paper/2411.07765/) (2024)
- [Envision3D: One Image to 3D with Anchor Views Interpolation](https://mlsanity.com/paper/2403.08902/) (2024)
- [Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects](https://mlsanity.com/paper/2504.08125/) (2025)
- [Scaling Sequence-to-Sequence Generative Neural Rendering](https://mlsanity.com/paper/2510.04236/) (2025)
- [Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories](https://mlsanity.com/paper/2604.09429/) (2026)
- [UniQueR: Unified Query-based Feedforward 3D Reconstruction](https://mlsanity.com/paper/2603.22851/) (2026)
- [Complete Gaussian Splats from a Single Image with Denoising Diffusion Models](https://mlsanity.com/paper/2508.21542/) (2025)
- [DT-NVS: Diffusion Transformers for Novel View Synthesis](https://mlsanity.com/paper/2511.08823/) (2025)
- [Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models](https://mlsanity.com/paper/2607.19120/) (2026)
- [Real-Time Position-Aware View Synthesis from Single-View Input](https://mlsanity.com/paper/2412.14005/) (2024)
- [Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views](https://mlsanity.com/paper/2505.11467/) (2025)
- [Novel View Synthesis using DDIM Inversion](https://mlsanity.com/paper/2508.10688/) (2025)
- [Inverse Image-Based Rendering for Light Field Generation from Single Images](https://mlsanity.com/paper/2510.20132/) (2025)
- [One View Is Enough: In-the-Wild Monocular Pretraining for Novel View Generation](https://mlsanity.com/paper/2603.23488/) (2026)

## Abstract (arXiv)

We propose NViST, a transformer-based model for efficient and generalizable novel-view synthesis from a single image for real-world scenes. In contrast to many methods that are trained on synthetic data, object-centred scenarios, or in a category-specific manner, NViST is trained on MVImgNet, a large-scale dataset of casually-captured real-world videos of hundreds of object categories with diverse backgrounds. NViST transforms image inputs directly into a radiance field, conditioned on camera parameters via adaptive layer normalisation. In practice, NViST exploits fine-tuned masked autoencoder (MAE) features and translates them to 3D output tokens via cross-attention, while addressing occlusions with self-attention. To move away from object-centred datasets and enable full scene synthesis, NViST adopts a 6-DOF camera pose model and only requires relative pose, dropping the need for canonicalization of the training data, which removes a substantial barrier to it being used on casually captured datasets. We show results on unseen objects and categories from MVImgNet and even generalization to casual phone captures. We conduct qualitative and quantitative evaluations on MVImgNet and ShapeNet to show that our model represents a step forward towards enabling true in-the-wild generalizable novel-view synthesis from a single image. Project webpage: https://wbjang.github.io/nvist_webpage.