mlsanityData to 7 Oct 2026

LLaNA: Large Language and NeRF Assistant

arXiv 2406.11840 ↗First posted 17 Jun 2024cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Meanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general-purpose NeRF-language assistant capable of performing new tasks such as NeRF captioning and Q&A. Notably, our method directly processes the weights of the NeRF's MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with no human intervention. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 86 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

71 of 86 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  8. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2023LERF: Language Embedded Radiance Fieldscited by 538
  17. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  18. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  19. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  20. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  21. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  22. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  23. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  24. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  25. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  26. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  27. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  28. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  29. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  30. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  31. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  32. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  33. 2020Predicting Neural Network Accuracy from Weightscited by 107
  34. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  35. 2023NerfAcc: Efficient Sampling Accelerates NeRFscited by 90
  36. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  37. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  38. 2023Permutation Equivariant Neural Functionalscited by 69

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  2. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  3. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  4. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embeddingcited by 1
  5. 2025Weight Space Representation Learning on Diverse NeRF Architecturescited by 0
  6. 2025Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?
  2. 2023RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  5. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  6. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
  7. 2024ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
  8. 2024TRINS: Towards Multimodal Language Models that Can Read