# PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes

arXiv: https://arxiv.org/abs/2512.24986 · HTML: https://mlsanity.com/paper/2512.24986/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code](https://mlsanity.com/paper/2403.01248/) (2024)
- [GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs](https://mlsanity.com/paper/2312.00093/) (2023)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents](https://mlsanity.com/paper/2402.05746/) (2024)
- [Octopus: Embodied Vision-Language Programmer from Environmental Feedback](https://mlsanity.com/paper/2310.08588/) (2023)
- [GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning](https://mlsanity.com/paper/2311.12631/) (2023)
- [Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Rendering](https://mlsanity.com/paper/2401.15318/) (2024)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [Dynamic Mesh-Aware Radiance Fields](https://mlsanity.com/paper/2309.04581/) (2023)
- [Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts](https://mlsanity.com/paper/2407.06842/) (2024)
- [Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2411.19233/) (2024)
- [PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation](https://mlsanity.com/paper/2506.01091/) (2025)

## Built on by (most cited first, 3 of 3)

- [NEWTON: Agentic Planning for Physically Grounded Video Generation](https://mlsanity.com/paper/2605.18396/) (2026)
- [i-PhysGaussian: Implicit Physical Simulation for 3D Gaussian Splatting](https://mlsanity.com/paper/2602.17117/) (2026)
- [A differentiable Lagrangian-coupled 3D Gaussian Splatting-SPH model for forward simulation and inverse analysis in solid mechanics](https://mlsanity.com/paper/2610.04336/) (2026)

## Abstract (arXiv)

Realistic visual simulations are omnipresent, yet their creation requires computing time, rendering, and expert animation knowledge. Open-vocabulary visual effects generation from text inputs emerges as a promising solution that can unlock immense creative potential. However, current pipelines lack both physical realism and effective language interfaces, requiring slow offline optimization. In contrast, PhysTalk takes a 3D Gaussian Splatting (3DGS) scene as input and translates arbitrary user prompts into real time, physics based, interactive 4D animations. A large language model (LLM) generates executable code that directly modifies 3DGS parameters through lightweight proxies and particle dynamics. Notably, PhysTalk is the first framework to couple 3DGS directly with a physics simulator without relying on time consuming mesh extraction. While remaining open vocabulary, this design enables interactive 3D Gaussian animation via collision aware, physics based manipulation of arbitrary, multi material objects. Finally, PhysTalk is train-free and computationally lightweight: this makes 4D animation broadly accessible and shifts these workflows from a "render and wait" paradigm toward an interactive dialogue with a modern, physics-informed pipeline.