# SplatCtrl: Perception-Action Coupling via Gaussian Scene Representations and Reactive Robot Control

arXiv: https://arxiv.org/abs/2607.08948 · HTML: https://mlsanity.com/paper/2607.08948/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 30 of 30)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [3D Gaussian Splatting as Markov Chain Monte Carlo](https://mlsanity.com/paper/2404.09591/) (2024)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting](https://mlsanity.com/paper/2409.10161/) (2024)
- [Motion Policy Networks](https://mlsanity.com/paper/2210.12209/) (2022)
- [Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps](https://mlsanity.com/paper/2403.02751/) (2024)
- [Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representation](https://mlsanity.com/paper/2408.14873/) (2024)
- [Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning](https://mlsanity.com/paper/2402.02500/) (2024)
- [3D Gaussian Splatting in Robotics: A Survey](https://mlsanity.com/paper/2410.12262/) (2024)
- [Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splatting](https://mlsanity.com/paper/2405.04378/) (2024)
- [Real-time Perception meets Reactive Motion Generation](https://mlsanity.com/paper/1703.03512/) (2017)
- [GS-Planner: A Gaussian-Splatting-based Planning Framework for Active High-Fidelity Reconstruction](https://mlsanity.com/paper/2405.10142/) (2024)
- [Neural MP: A Generalist Neural Motion Planner](https://mlsanity.com/paper/2409.05864/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Stochastic Poisson Surface Reconstruction](https://mlsanity.com/paper/2206.15236/) (2022)
- [SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps](https://mlsanity.com/paper/2409.09868/) (2024)
- [EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation](https://mlsanity.com/paper/2310.06751/) (2023)
- [Accurate Gaussian-Process-based Distance Fields with applications to Echolocation and Mapping](https://mlsanity.com/paper/2302.13005/) (2023)
- [Distance and Collision Probability Estimation from Gaussian Surface Models](https://mlsanity.com/paper/2402.00186/) (2024)
- [Isotropic Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2403.14244/) (2024)
- [FOCI: Trajectory Optimization on Gaussian Splats](https://mlsanity.com/paper/2505.08510/) (2025)
- [Towards Feasible Dynamic Grasping: Leveraging Gaussian Process Distance Field, SE(3) Equivariance and Riemannian Mixture Models](https://mlsanity.com/paper/2311.02576/) (2023)
- [Constrained Passive Interaction Control: Leveraging Passivity and Safety for Robot Manipulators](https://mlsanity.com/paper/2403.09853/) (2024)

## Abstract (arXiv)

Robotic manipulators excel in structured environments but face substantial challenges in unstructured and dynamic settings. This paper presents SplatCtrl, a unified framework for real-time scene reconstruction and reactive robot motion generation to enable collision-free robotic arm control in previously unseen and continuously changing environments. Building on 3D Gaussian Splatting (3D-GS), we introduce a hybrid voxel-based filtering and dynamic Gaussian relocation strategy that supports efficient scene reconstruction from RGB-D streams while accommodating environmental changes. For safe and reactive control, we further propose a method for deriving continuous signed distance functions from isotropic Gaussians, providing stable and differentiable collision probability estimates that bridge classical distance fields with the modern implicit representation. These continuous distance metrics are incorporated into control barrier functions, resulting in a unified perception-action coupling framework that supports smooth and reliable real-time motion generation in response to scene changes. Experimental validation in simulation, on physical robot, and within shared human-robot workspace demonstrates the framework's effectiveness, achieving integrated scene reconstruction and reactive control in uncertain, and dynamic environments.