mlsanityData to 7 Oct 2026

PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis

arXiv 2402.17986 ↗First posted 28 Feb 2024cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper considers the problem of generative novel view synthesis (GNVS), generating novel, plausible views of a scene given a limited number of known views. Here, we propose a set-based generative model that can simultaneously generate multiple, self-consistent new views, conditioned on any number of views. Our approach is not limited to generating a single image at a time and can condition on a variable number of views. As a result, when generating a large number of views, our method is not restricted to a low-order autoregressive generation approach and is better able to maintain generated image quality over large sets of images. We evaluate our model on standard NVS datasets and show that it outperforms the state-of-the-art image-based GNVS baselines. Further, we show that the model is capable of generating sets of views that have no natural sequential ordering, like loops and binocular trajectories, and significantly outperforms other methods on such tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 55 of its 70 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

55 of 70 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2022Video Diffusion Modelscited by 2,280
38 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  6. 2017Deep Setscited by 1,394
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2022Diffusion Models in Vision: A Surveycited by 1,008
  10. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  12. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  13. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  14. 2021Neural Fields in Visual Computing and Beyondcited by 539
  15. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  16. 2022Novel View Synthesis with Diffusion Modelscited by 299
  17. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  18. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  19. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  20. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  21. 2019PointNetLK: Robust & Efficient Point Cloud Registration using PointNetcited by 231
  22. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  23. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  24. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  25. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 158
  26. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  27. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 142
  28. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  29. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  30. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  31. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Datacited by 113
  32. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  33. 2022Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Cloudscited by 102
  34. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  35. 2021SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Datacited by 78
  36. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  37. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  38. 2019Deep Set Prediction Networkscited by 71

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 38
  2. 2024Novel View Synthesis with Pixel-Space Diffusion Modelscited by 16
  3. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  4. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  5. 2025CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesiscited by 5
  6. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesiscited by 4
  7. 2025ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 1
  8. 2025Learn Your Scales: Towards Scale-Consistent Generative Novel View Synthesiscited by 0
  9. 2026AdaptiveSplat:Texture Aware Controllable 3D Gaussian Allocation for Feed-Forward Reconstructioncited by 0
  10. 2026GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generationcited by 0