mlsanityData to 7 Oct 2026

Connecting NeRFs, Images, and Text

arXiv 2404.07993 ↗First posted 11 Apr 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Neural Radiance Fields (NeRFs) have emerged as a standard framework for representing 3D scenes and objects, introducing a novel data type for information exchange and storage. Concurrently, significant progress has been made in multimodal representation learning for text and image data. This paper explores a novel research direction that aims to connect the NeRF modality with other modalities, similar to established methodologies for images and text. To this end, we propose a simple framework that exploits pre-trained models for NeRF representations alongside multimodal models for text and image processing. Our framework learns a bidirectional mapping between NeRF embeddings and those obtained from corresponding images and text. This mapping unlocks several novel and useful applications, including NeRF zero-shot classification and NeRF retrieval from images or text.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

45 of 54 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
33 more
  1. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  4. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  12. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  13. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  14. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  15. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  16. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  17. 2020Predicting Neural Network Accuracy from Weightscited by 107
  18. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  19. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  20. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  21. 2023Permutation Equivariant Neural Functionalscited by 69
  22. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  23. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  24. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  25. 2023Neural Functional Transformerscited by 50
  26. 2022NeRF-RPN: A general framework for object detection in NeRFscited by 46
  27. 2019Neural Network Branching for Neural Network Verificationcited by 45
  28. 2024Universal Neural Functionalscited by 25
  29. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  30. 2022ScanNeRF: a Scalable Benchmark for Neural Radiance Fieldscited by 14
  31. 2021Generating Adversarial Examples with Graph Neural Networkscited by 12
  32. 2023Neural Radiance Fields: Past, Present, and Futurecited by 8
  33. 2023Neural Processing of Tri-Plane Hybrid Neural Fieldscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  2. 2025UniGS: Unified Language-Image-3D Pretraining with Gaussian Splattingcited by 5
  3. 2026RadarSim: Simulating Single-Chip Radar via Multimodal Neural Fieldscited by 2
  4. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embeddingcited by 1
  5. 2025Weight Space Representation Learning on Diverse NeRF Architecturescited by 0
  6. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Trainingcited by 0