# 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2503.10437 · HTML: https://mlsanity.com/paper/2503.10437/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 60 of its 71 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 60)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)

## Built on by (most cited first, 30 of 35)

- [LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS](https://mlsanity.com/paper/2507.07136/) (2025)
- [LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding](https://mlsanity.com/paper/2505.12253/) (2025)
- [SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fields](https://mlsanity.com/paper/2506.09565/) (2025)
- [Quantifying and Alleviating Co-Adaptation in Sparse-View 3D Gaussian Splatting](https://mlsanity.com/paper/2508.12720/) (2025)
- [Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting](https://mlsanity.com/paper/2503.22204/) (2025)
- [Mitigating Semantic Collapse in Partially Relevant Video Retrieval](https://mlsanity.com/paper/2510.27432/) (2025)
- [DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling](https://mlsanity.com/paper/2512.03000/) (2025)
- [4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2512.05060/) (2025)
- [Consistent Instance Field for Dynamic Scene Understanding](https://mlsanity.com/paper/2512.14126/) (2025)
- [LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents](https://mlsanity.com/paper/2512.17445/) (2025)
- [4D Synchronized Fields: Motion-Language Gaussian Splatting for Temporal Scene Understanding](https://mlsanity.com/paper/2603.14301/) (2026)
- [LangFlash: Feed-forward 3D Language Gaussian Splatting from Sparse Unposed Images](https://mlsanity.com/paper/2605.23287/) (2026)
- [MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model](https://mlsanity.com/paper/2606.08288/) (2026)
- [SIMSplat: Language-Aligned 4D Gaussian Splatting for Driving Scenario Generation](https://mlsanity.com/paper/2510.02469/) (2025)
- [Mono4DEditor: Text-Driven 4D Scene Editing from Monocular Video via Point-Level Localization of Language-Embedded Gaussians](https://mlsanity.com/paper/2510.09438/) (2025)
- [TIBR4D: Tracing-Guided Iterative Boundary Refinement for Efficient 4D Gaussian Segmentation](https://mlsanity.com/paper/2602.08540/) (2026)
- [IRIS-SLAM: Unified Geo-Instance Representations for Robust Semantic Localization and Mapping](https://mlsanity.com/paper/2602.18709/) (2026)
- [PanopticQuery: Unified Query-Time Reasoning for 4D Scenes](https://mlsanity.com/paper/2604.05638/) (2026)
- [Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images](https://mlsanity.com/paper/2607.01633/) (2026)
- [CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting](https://mlsanity.com/paper/2505.18306/) (2025)
- [WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videos](https://mlsanity.com/paper/2602.20556/) (2026)
- [ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors](https://mlsanity.com/paper/2603.04338/) (2026)
- [Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding](https://mlsanity.com/paper/2604.01678/) (2026)
- [Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors](https://mlsanity.com/paper/2605.25909/) (2026)
- [Do Gaussian Scenes Contain Enough Structure for Intrinsic Segmentation?](https://mlsanity.com/paper/2606.18623/) (2026)
- [Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation](https://mlsanity.com/paper/2606.22197/) (2026)
- [SemDynReg: Semantics-Guided Deformation Regularization for Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2606.28656/) (2026)
- [DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming](https://mlsanity.com/paper/2606.28840/) (2026)
- [SAD-GS: Learning Reliable 3D Semantic Gaussian Fields via Dynamic Geo-Semantic Anchoring](https://mlsanity.com/paper/2606.29376/) (2026)
- [MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians](https://mlsanity.com/paper/2607.28300/) (2026)

## Abstract (arXiv)

Learning 4D language fields to enable time-sensitive, open-ended language queries in dynamic scenes is essential for many real-world applications. While LangSplat successfully grounds CLIP features into 3D Gaussian representations, achieving precision and efficiency in 3D static scenes, it lacks the ability to handle dynamic 4D fields as CLIP, designed for static image-text tasks, cannot capture temporal dynamics in videos. Real-world environments are inherently dynamic, with object semantics evolving over time. Building a precise 4D language field necessitates obtaining pixel-aligned, object-wise video features, which current vision models struggle to achieve. To address these challenges, we propose 4D LangSplat, which learns 4D language fields to handle time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes efficiently. 4D LangSplat bypasses learning the language field from vision features and instead learns directly from text generated from object-wise video captions via Multimodal Large Language Models (MLLMs). Specifically, we propose a multimodal object-wise video prompting method, consisting of visual and text prompts that guide MLLMs to generate detailed, temporally consistent, high-quality captions for objects throughout a video. These captions are encoded using a Large Language Model into high-quality sentence embeddings, which then serve as pixel-aligned, object-specific feature supervision, facilitating open-vocabulary text queries through shared embedding spaces. Recognizing that objects in 4D scenes exhibit smooth transitions across states, we further propose a status deformable network to model these continuous changes over time effectively. Our results across multiple benchmarks demonstrate that 4D LangSplat attains precise and efficient results for both time-sensitive and time-agnostic open-vocabulary queries.