mlsanityData to 14 Oct 2026

VQ3D: Learning a 3D-Aware Generative Model on ImageNet

arXiv 2302.06833 ↗First posted 14 Feb 2023cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars. However, these models struggle on larger, more complex datasets. To model diverse and unconstrained image collections such as ImageNet, we present VQ3D, which introduces a NeRF-based decoder into a two-stage vector-quantized autoencoder. Our Stage 1 allows for the reconstruction of an input image and the ability to change the camera position around the image, and our Stage 2 allows for the generation of new 3D scenes. VQ3D is capable of generating and reconstructing 3D-aware images from the 1000-class ImageNet dataset of 1.2 million training images. We achieve an ImageNet generation FID score of 16.8, compared to 69.8 for the next best baseline method.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

43 of 54 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
31 more
  1. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  2. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  3. 2021Vision Transformers for Dense Predictioncited by 1,803
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  7. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  10. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  11. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  14. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  15. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  16. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  17. 2022Kubric: A scalable dataset generatorcited by 381
  18. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  19. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  20. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  21. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  22. 2015A Large-Scale Car Dataset for Fine-Grained Categorization and Verificationcited by 130
  23. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 112
  24. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  25. 2021Unsupervised Discovery of Object Radiance Fieldscited by 101
  26. 2022GIRAFFE HD: A High-Resolution 3D-aware Generative Modelcited by 86
  27. 2021LOLNeRF: Learn from One Lookcited by 84
  28. 2022Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translationcited by 73
  29. 2020Lifting 2D StyleGAN for 3D-Aware Face Generationcited by 64
  30. 2022Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Imagescited by 39
  31. 2018Learning View Priors for Single-view 3D Reconstructioncited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 10; 2025: 6; 2026: 2.

32 papers.

Sort:

By area:

  1. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  2. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 158
  3. 2023WonderJourney: Going from Anywhere to Everywherecited by 158
  4. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 123
  5. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 111
  6. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 92
  7. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesiscited by 84
  8. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 77
  9. 2025Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priorscited by 72
  10. 20233D-aware Image Generation using 2D Diffusion Modelscited by 69
  11. 2023Online Clustered Codebookcited by 55
  12. 2023GINA-3D: Learning to Generate Implicit Neural Assets in the Wildcited by 25
20 more
  1. 2024Real3D: Scaling Up Large Reconstruction Models with Real-World Imagescited by 24
  2. 2025CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Controlcited by 21
  3. 2023A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shotcited by 19
  4. 2023NViST: In the Wild New View Synthesis from a Single Image with Transformerscited by 19
  5. 2024ViewFusion: Towards Multi-View Consistency via Interpolated Denoisingcited by 14
  6. 2024Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fieldscited by 13
  7. 2024Neural Radiance Field-based Visual Rendering: A Comprehensive Reviewcited by 12
  8. 2025Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversioncited by 10
  9. 2023WildFusion: Learning 3D-Aware Latent Diffusion Models in View Spacecited by 8
  10. 2023Boosting3D: High-Fidelity Image-to-3D by Boosting 2D Diffusion Prior to 3D Prior with Progressive Learningcited by 7
  11. 2024Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAEcited by 7
  12. 2024DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Modelcited by 7
  13. 2024G3DR: Generative 3D Reconstruction in ImageNetcited by 5
  14. 2024Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolutioncited by 5
  15. 2024Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimationcited by 3
  16. 2025F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Aggregative Gaussian Splattingcited by 3
  17. 2023Novel View Synthesis with View-Dependent Effects from a Single Imagecited by 2
  18. 2025DT-NVS: Diffusion Transformers for Novel View Synthesiscited by 1
  19. 2026ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margincited by 0
  20. 2026CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebookcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models
  2. 2026Class-Partitioned VQ-VAE and Latent Flow Matching for Point Cloud Scene Generation
  3. 2023VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
  4. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE
  5. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
  6. 2026GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation
  7. 20243D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing
  8. 2022Deep Generative Models on 3D Representations: A Survey