mlsanityData to 7 Oct 2026

ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation

arXiv 2403.08321 ↗First posted 13 Mar 2024cs.RO cs.CVcited by 106

Not on the map. Why?

Abstract (arXiv)

Performing language-conditioned robotic manipulation tasks in unstructured environments is highly demanded for general intelligent robots. Conventional robotic manipulation methods usually learn semantic representation of the observation for action prediction, which ignores the scene-level spatiotemporal dynamics for human goal completion. In this paper, we propose a dynamic Gaussian Splatting method named ManiGaussian for multi-task robotic manipulation, which mines scene dynamics via future scene reconstruction. Specifically, we first formulate the dynamic Gaussian Splatting framework that infers the semantics propagation in the Gaussian embedding space, where the semantic representation is leveraged to predict the optimal robot action. Then, we build a Gaussian world model to parameterize the distribution in our dynamic Gaussian Splatting framework, which provides informative supervision in the interactive environment via future scene reconstruction. We evaluate our ManiGaussian on 10 RLBench tasks with 166 variations, and the results demonstrate our framework can outperform the state-of-the-art methods by 13.1% in average success rate. Project page: https://guanxinglu.github.io/ManiGaussian/.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 77 of its 84 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

77 of 84 references.

References · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Segment Anythingcited by 8,158
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2023Mastering Diverse Domains through World Modelscited by 1,193
  10. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2020Mastering Atari with Discrete World Modelscited by 990
38 more
  1. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  2. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  13. 2023LERF: Language Embedded Radiance Fieldscited by 538
  14. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  15. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  16. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  17. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 470
  18. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  19. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  20. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  21. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  22. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  23. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  24. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  25. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  26. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  27. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  28. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  29. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  30. 2021Mastering Atari Games with Limited Datacited by 235
  31. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  32. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  33. 2024A Survey on 3D Gaussian Splattingcited by 221
  34. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  35. 2023COLMAP-Free 3D Gaussian Splattingcited by 193
  36. 2023GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesiscited by 170
  37. 2022Masked World Models for Visual Controlcited by 169
  38. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 52; 2026: 33.

106 papers.

Sort:

By area:

  1. 2024Recent Advances in 3D Gaussian Splattingcited by 130
  2. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 77
  3. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 65
  6. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  7. 2024RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulatorcited by 50
  8. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  9. 2024GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modelingcited by 44
  10. 2024DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priorscited by 40
  11. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 37
  12. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
38 more
  1. 2025ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splattingcited by 35
  2. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  3. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  4. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  5. 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 30
  6. 2025Decompositional Neural Scene Reconstruction with Generative Diffusion Priorcited by 30
  7. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  8. 2024Momentum-GS: Momentum Gaussian Self-Distillation for High-Quality Large Scene Reconstructioncited by 27
  9. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  10. 20253D Scene Generation: A Surveycited by 24
  11. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  12. 2024Neural Fields in Robotics: A Surveycited by 21
  13. 2025Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulationcited by 18
  14. 2024LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understandingcited by 17
  15. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  16. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Graspingcited by 15
  17. 2024AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulationcited by 15
  18. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  19. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  20. 2025FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulationcited by 13
  21. 2025Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracingcited by 13
  22. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 13
  23. 2026VLANeXt: Recipes for Building Strong VLA Modelscited by 12
  24. 2026WAM4D: Fast 4D World Action Model via Spatial Register Tokenscited by 12
  25. 2025Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Frameworkcited by 11
  26. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  27. 2024Next Best Sense: Guiding Vision and Touch with FisherRF for 3D Gaussian Splattingcited by 10
  28. 2025TACO: Taming Diffusion for in-the-wild Video Amodal Completioncited by 10
  29. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 10
  30. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  31. 2025IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigationcited by 10
  32. 2026Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulationcited by 10
  33. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  34. 2025TriVLA: A Triple-System-Based Unified Vision-Language-Action Model with Episodic World Modeling for General Robot Controlcited by 9
  35. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learningcited by 8
  36. 2025PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesiscited by 8
  37. 2026AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Modelscited by 7
  38. 2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation
  2. 2025Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting ↗
  3. 2026Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation
  4. 2024Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics
  5. 2025High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting
  6. 2025ArtGS:3D Gaussian Splatting for Interactive Visual-Physical Modeling and Manipulation of Articulated Objects
  7. 2026World Models for Robotic Manipulation: A Survey
  8. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation