mlsanityData to 14 Oct 2026

Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training

arXiv 2504.13995 ↗First posted 18 Apr 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advances in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities in understanding both images and 3D data, yet these modalities face inherent limitations in comprehensively representing object geometry and appearance. Neural Radiance Fields (NeRFs) have emerged as a promising alternative, encoding both geometric and photorealistic properties within the weights of a simple Multi-Layer Perceptron (MLP). This work investigates the feasibility and effectiveness of ingesting NeRFs into an MLLM. We introduce LLaNA, the first MLLM able to perform new tasks such as NeRF captioning and Q&A, by directly processing the weights of a NeRF's MLP. Notably, LLaNA is able to extract information about the represented objects without the need to render images or materialize 3D data structures. In addition, we build the first large-scale NeRF-language dataset, composed by more than 300K NeRFs trained on ShapeNet and Objaverse, with paired textual annotations that enable various NeRF-language tasks. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that directly processing NeRF weights leads to better performance on NeRF-Language tasks compared to approaches that rely on either 2D or 3D representations derived from NeRFs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 82 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

69 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaNA: Large Language and NeRF Assistant17 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
38 more
  1. 2015Fast R-CNNcited by 2,933
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  7. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  8. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  14. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2023LERF: Language Embedded Radiance Fieldscited by 543
  17. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  18. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  19. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  20. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  21. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  23. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  24. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  25. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  26. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  27. 2022From data to functa: Your data point is a function and you can treat it like onecited by 187
  28. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 155
  29. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  30. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  31. 2020Predicting Neural Network Accuracy from Weightscited by 110
  32. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 107
  33. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  34. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  35. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  36. 2023Permutation Equivariant Neural Functionalscited by 70
  37. 2023Deep Learning on Implicit Neural Representations of Shapescited by 63
  38. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60

50 most cited of 67 All 67 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
  2. 2024M2DNeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields
  3. 2023Strata-NeRF : Neural Radiance Fields for Stratified Scenes
  4. 2024ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
  5. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
  6. 2023ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context
  7. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  8. 2023Learning a Diffusion Prior for NeRFs