mlsanityData to 7 Oct 2026

EscherNet: A Generative Model for Scalable View Synthesis

arXiv 2402.03908 ↗First posted 6 Feb 2024cs.CVcited by 94

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce EscherNet, a multi-view conditioned diffusion model for view synthesis. EscherNet learns implicit and generative 3D representations coupled with a specialised camera positional encoding, allowing precise and continuous relative control of the camera transformation between an arbitrary number of reference and target views. EscherNet offers exceptional generality, flexibility, and scalability in view synthesis -- it can generate more than 100 consistent target views simultaneously on a single consumer-grade GPU, despite being trained with a fixed number of 3 reference views to 3 target views. As a result, EscherNet not only addresses zero-shot novel view synthesis, but also naturally unifies single- and multi-image 3D reconstruction, combining these diverse tasks into a single, cohesive framework. Our extensive experiments demonstrate that EscherNet achieves state-of-the-art performance in multiple benchmarks, even when compared to methods specifically tailored for each individual problem. This remarkable versatility opens up new directions for designing scalable neural architectures for 3D vision. Project page: https://kxhit.github.io/EscherNet.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 54 of its 64 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

54 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  2. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  3. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models4 Oct
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  5. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  6. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  7. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  9. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  10. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  11. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views29 Nov
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
2 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  2. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
28 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  8. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  11. 2020Convolutional Occupancy Networkscited by 760
  12. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  13. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  14. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  15. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  16. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  17. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  18. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  19. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  20. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  21. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  22. 2023FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularizationcited by 310
  23. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  24. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  25. 2022Variable Bitrate Neural Fieldscited by 116
  26. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
  27. 2023Patched Denoising Diffusion Models For High-Resolution Image Synthesiscited by 45
  28. 2022RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesiscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 38; 2026: 30.

94 papers.

Sort:

By area:

  1. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  2. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  3. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  4. 2025Cameras as Relative Positional Encodingcited by 94
  5. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  6. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  7. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  8. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  9. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  10. 2024One Diffusion to Generate Them Allcited by 41
  11. 20243DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Viewscited by 37
  12. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
38 more
  1. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 34
  2. 2024Video Depth without Video Modelscited by 28
  3. 2024IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminationscited by 21
  4. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20
  5. 2024Hash3D: Training-free Acceleration for 3D Generationcited by 19
  6. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  7. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergycited by 16
  8. 2026MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generationcited by 16
  9. 2024SplatFormer: Point Transformer for Robust 3D Gaussian Splattingcited by 15
  10. 2025PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Modelcited by 15
  11. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13
  12. 2025Edit360: 2D Image Edits to 3D Assets from Any Anglecited by 13
  13. 20253DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Modelscited by 12
  14. 2025Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesiscited by 12
  15. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  16. 2026ReRoPE: Repurposing RoPE for Relative Camera Controlcited by 12
  17. 2024Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimationcited by 10
  18. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Modelscited by 10
  19. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 10
  20. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 10
  21. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
  22. 2024HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Modelcited by 9
  23. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generationcited by 9
  24. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  25. 2024Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generationscited by 9
  26. 2025MVGBench: Comprehensive Benchmark for Multi-view Generation Modelscited by 9
  27. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  28. 2025EgoX: Egocentric Video Generation from a Single Exocentric Videocited by 8
  29. 2024MVDiff: Scalable and Flexible Multi-View Diffusion for 3D Object Reconstruction from Single-Viewcited by 6
  30. 2025ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priorscited by 6
  31. 2025LVT: Large-Scale Scene Reconstruction via Local View Transformerscited by 6
  32. 2025WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusioncited by 6
  33. 2026Stream3D: Sequential Multi-View 3D Generation via Evidential Memorycited by 6
  34. 2025CoreEditor: Correspondence-constrained Diffusion for Consistent 3D Editingcited by 5
  35. 2025CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesiscited by 5
  36. 2026Scaling View Synthesis Transformerscited by 5
  37. 2026BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregressioncited by 5
  38. 2026Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillationcited by 5

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Novel View Synthesis with Pixel-Space Diffusion Models
  2. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
  3. 2024MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction
  4. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models
  5. 2024NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images
  6. 2026BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model
  7. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Models
  8. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models