# NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting

arXiv: https://arxiv.org/abs/2507.12621 · HTML: https://mlsanity.com/paper/2507.12621/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 47 of its 75 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding](https://mlsanity.com/paper/2406.02058/) (2024)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [3D Gaussian as a New Era: A Survey](https://mlsanity.com/paper/2402.07181/) (2024)
- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15624/) (2024)
- [StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting](https://mlsanity.com/paper/2403.07807/) (2024)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations](https://mlsanity.com/paper/1908.00407/) (2019)
- [DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualization](https://mlsanity.com/paper/2204.06504/) (2022)
- [A Generative Model for Volume Rendering](https://mlsanity.com/paper/1710.09545/) (2017)

## Built on by (most cited first, 20 of 20)

- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents](https://mlsanity.com/paper/2603.29139/) (2026)
- [TexGS-VolVis: Expressive Scene Editing for Volume Visualization via Textured Gaussian Splatting](https://mlsanity.com/paper/2507.13586/) (2025)
- [An Evaluation-Centric Paradigm for Scientific Visualization Agents](https://mlsanity.com/paper/2509.15160/) (2025)
- [TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization](https://mlsanity.com/paper/2603.25063/) (2026)
- [Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Tasks](https://mlsanity.com/paper/2605.21825/) (2026)
- [ECoNGS: Efficient Compressive Neural Gaussian Splats for Volume Visualization](https://mlsanity.com/paper/2607.18466/) (2026)
- [Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization](https://mlsanity.com/paper/2604.27996/) (2026)
- [HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization](https://mlsanity.com/paper/2606.26614/) (2026)
- [Efficient Compression of Structured and Unstructured Volumes via Learned 3D Gaussian Representation](https://mlsanity.com/paper/2607.01164/) (2026)
- [Animating Petascale Time-varying Data on Commodity Hardware with LLM-assisted Scripting](https://mlsanity.com/paper/2603.07053/) (2026)
- [SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualization](https://mlsanity.com/paper/2606.05525/) (2026)
- [Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration](https://mlsanity.com/paper/2607.16352/) (2026)
- [Toward Reliable Scientific Visualization Pipeline Construction with Structure-Aware Retrieval-Augmented LLMs](https://mlsanity.com/paper/2603.16057/) (2026)
- [Topo-GS: Continuous Volumetric Embedding of High-Dimensional Data via Topological Gaussian Splatting](https://mlsanity.com/paper/2605.17011/) (2026)
- [Lossless-INR: Lossless Volumetric Implicit Neural Representations](https://mlsanity.com/paper/2607.18150/) (2026)
- [EVOLVE: Efficient Learned Volume Compression with Variable-Rate Encoding on a Cross-Domain Database](https://mlsanity.com/paper/2607.18187/) (2026)
- [Multi-Agent Discovery and Resource-Aware Autonomous Exploration of Scientific Datasets](https://mlsanity.com/paper/2608.22045/) (2026)
- [MedVA: An End-to-End Neuro-Symbolic Agentic System for Medical Volume Visualization](https://mlsanity.com/paper/2609.14874/) (2026)
- [VETR: Execution-Grounded Self-Improving Agent for Symbolic Graphics Reasoning](https://mlsanity.com/paper/2610.13098/) (2026)

## Abstract (arXiv)

Traditional volume visualization (VolVis) methods, like direct volume rendering, suffer from rigid transfer function designs and high computational costs. Although novel view synthesis approaches enhance rendering efficiency, they require additional learning effort for non-experts and lack support for semantic-level interaction. To bridge this gap, we propose NLI4VolVis, an interactive system that enables users to explore, query, and edit volumetric scenes using natural language. NLI4VolVis integrates multi-view semantic segmentation and vision-language models to extract and understand semantic components in a scene. We introduce a multi-agent large language model architecture equipped with extensive function-calling tools to interpret user intents and execute visualization tasks. The agents leverage external tools and declarative VolVis commands to interact with the VolVis engine powered by 3D editable Gaussians, enabling open-vocabulary object querying, real-time scene editing, best-view selection, and 2D stylization. We validate our system through case studies and a user study, highlighting its improved accessibility and usability in volumetric data exploration. We strongly recommend readers check our case studies, demo video, and source code at https://nli4volvis.github.io/.