mlsanityData to 7 Oct 2026

HyperFields: Towards Zero-Shot Generation of NeRFs from Text

arXiv 2310.17075 ↗First posted 26 Oct 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce HyperFields, a method for generating text-conditioned Neural Radiance Fields (NeRFs) with a single forward pass and (optionally) some fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF distillation training, which distills scenes encoded in individual NeRFs into one dynamic hypernetwork. These techniques enable a single network to fit over a hundred unique scenes. We further demonstrate that HyperFields learns a more general map between text and NeRFs, and consequently is capable of predicting novel in-distribution and out-of-distribution scenes -- either zero-shot or with a few finetuning steps. Finetuning HyperFields benefits from accelerated convergence thanks to the learned general map, and is capable of synthesizing novel scenes 5 to 10 times faster than existing neural optimization-based methods. Our ablation experiments show that both the dynamic architecture and NeRF distillation are critical to the expressivity of HyperFields.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

37 of 48 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  9. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  11. 2018On the Spectral Bias of Neural Networkscited by 1,321
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
25 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  3. 2016HyperNetworkscited by 1,003
  4. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  5. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  6. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  7. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  9. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  10. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  11. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  12. 2021Neural Fields in Visual Computing and Beyondcited by 539
  13. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  14. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  15. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  16. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  17. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  18. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  19. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  20. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  21. 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
  22. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  23. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  24. 2021Stylizing 3D Scene via Implicit Representation and HyperNetworkcited by 103
  25. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 3.

13 papers.

Sort:

By area:

  1. 2024Consistency Models Made Easycited by 140
  2. 20233D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillationcited by 34
  3. 2024ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillationcited by 27
  4. 2024AToM: Amortized Text-to-Mesh using 2D Diffusioncited by 18
  5. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  6. 2024A Survey On Text-to-3D Contents Generation In The Wildcited by 13
  7. 2024Text2Robot: Evolutionary Robot Design from Text Descriptionscited by 8
  8. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generationcited by 7
  9. 2024HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectoriescited by 4
  10. 2025GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfercited by 2
  11. 2024VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchificationcited by 0
  12. 2024Any-to-3D Generation via Hybrid Diffusion Supervisioncited by 0
1 more
  1. 2025HyperDiffusionFields (HyDiF): Diffusion-Guided Hypernetworks for Learning Implicit Molecular Neural Fieldscited by 0