# DT-NVS: Diffusion Transformers for Novel View Synthesis

arXiv: https://arxiv.org/abs/2511.08823 · HTML: https://mlsanity.com/paper/2511.08823/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 22 of its 79 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 22 of 22)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2312.02201/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model](https://mlsanity.com/paper/2311.09217/) (2023)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image](https://mlsanity.com/paper/2310.17994/) (2023)
- [Multiview Compressive Coding for 3D Reconstruction](https://mlsanity.com/paper/2301.08247/) (2023)
- [Progressive Distillation for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2202.00512/) (2022)
- [Explicit Correspondence Matching for Generalizable Neural Radiance Fields](https://mlsanity.com/paper/2304.12294/) (2023)
- [VQ3D: Learning a 3D-Aware Generative Model on ImageNet](https://mlsanity.com/paper/2302.06833/) (2023)
- [NViST: In the Wild New View Synthesis from a Single Image with Transformers](https://mlsanity.com/paper/2312.08568/) (2023)
- [Denoising Diffusion via Image-Based Rendering](https://mlsanity.com/paper/2402.03445/) (2024)
- [MorpheuS: Neural Dynamic 360deg Surface Reconstruction from Monocular RGB-D Video](https://mlsanity.com/paper/2312.00778/) (2023)

## Built on by (most cited first, 1 of 1)

- [Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models](https://mlsanity.com/paper/2607.19120/) (2026)

## Abstract (arXiv)

Generating novel views of a natural scene, e.g., every-day scenes both indoors and outdoors, from a single view is an under-explored problem, even though it is an organic extension to the object-centric novel view synthesis. Existing diffusion-based approaches focus rather on small camera movements in real scenes or only consider unnatural object-centric scenes, limiting their potential applications in real-world settings. In this paper we move away from these constrained regimes and propose a 3D diffusion model trained with image-only losses on a large-scale dataset of real-world, multi-category, unaligned, and casually acquired videos of everyday scenes. We propose DT-NVS, a 3D-aware diffusion model for generalized novel view synthesis that exploits a transformer-based architecture backbone. We make significant contributions to transformer and self-attention architectures to translate images to 3d representations, and novel camera conditioning strategies to allow training on real-world unaligned datasets. In addition, we introduce a novel training paradigm swapping the role of reference frame between the conditioning image and the sampled noisy input. We evaluate our approach on the 3D task of generalized novel view synthesis from a single input image and show improvements over state-of-the-art 3D aware diffusion models and deterministic approaches, while generating diverse outputs.