# SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum

arXiv: https://arxiv.org/abs/2412.16346 · HTML: https://mlsanity.com/paper/2412.16346/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [gsplat: An Open-Source Library for Gaussian Splatting](https://mlsanity.com/paper/2409.06765/) (2024)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation](https://mlsanity.com/paper/2408.11812/) (2024)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control](https://mlsanity.com/paper/2103.02142/) (2021)
- [SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting](https://mlsanity.com/paper/2409.10161/) (2024)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis](https://mlsanity.com/paper/2301.08556/) (2023)
- [Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps](https://mlsanity.com/paper/2403.02751/) (2024)
- [Learning to Fly by Crashing](https://mlsanity.com/paper/1704.05588/) (2017)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories](https://mlsanity.com/paper/1712.02402/) (2017)
- [Demonstrating Agile Flight from Pixels without State Estimation](https://mlsanity.com/paper/2406.12505/) (2024)
- [FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation](https://mlsanity.com/paper/1905.11377/) (2019)
- [Learning a Single Near-hover Position Controller for Vastly Different Quadcopters](https://mlsanity.com/paper/2209.09232/) (2022)
- [A Learning-based Quadcopter Controller with Extreme Adaptation](https://mlsanity.com/paper/2409.12949/) (2024)
- [Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance](https://mlsanity.com/paper/2405.10391/) (2024)
- [Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks](https://mlsanity.com/paper/2406.15149/) (2024)
- [Accurate Tracking of Aggressive Quadrotor Trajectories using Incremental Nonlinear Dynamic Inversion and Differential Flatness](https://mlsanity.com/paper/1809.04048/) (2018)
- [Demonstration-Efficient Guided Policy Search via Imitation of Robust Tube MPC](https://mlsanity.com/paper/2109.09910/) (2021)
- [Pegasus Simulator: An Isaac Sim Framework for Multiple Aerial Vehicles Simulation](https://mlsanity.com/paper/2307.05263/) (2023)

## Built on by (most cited first, 16 of 16)

- [GRaD-Nav++: Vision-Language Model Enabled Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics](https://mlsanity.com/paper/2506.14009/) (2025)
- [EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device](https://mlsanity.com/paper/2509.17430/) (2025)
- [GRaD-Nav: Efficiently Learning Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics](https://mlsanity.com/paper/2503.03984/) (2025)
- [π, But Make It Fly: Physics-Guided Transfer of VLA Models to Aerial Manipulation](https://mlsanity.com/paper/2603.25038/) (2026)
- [SINGER: An Onboard Generalist Vision-Language Navigation Policy for Drones](https://mlsanity.com/paper/2509.18610/) (2025)
- [Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation](https://mlsanity.com/paper/2510.11689/) (2025)
- [ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles](https://mlsanity.com/paper/2602.11575/) (2026)
- [Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning](https://mlsanity.com/paper/2505.14938/) (2025)
- [LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World](https://mlsanity.com/paper/2606.01458/) (2026)
- [UAVTwin: Neural Digital Twins for UAVs using Gaussian Splatting](https://mlsanity.com/paper/2504.02158/) (2025)
- [FalconWing: An Ultra-Light Indoor Fixed-Wing UAV Platform for Vision-Based Autonomy](https://mlsanity.com/paper/2505.01383/) (2025)
- [SketchPlan: Diffusion Based Drone Planning From Human Sketches](https://mlsanity.com/paper/2510.03545/) (2025)
- [GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields](https://mlsanity.com/paper/2604.05062/) (2026)
- [FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking](https://mlsanity.com/paper/2606.29783/) (2026)
- [Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation](https://mlsanity.com/paper/2609.36520/) (2026)
- [ClearNav: Perception-Aligned Quadrotor Flight Using Differential-Physics](https://mlsanity.com/paper/2610.17419/) (2026)

## Abstract (arXiv)

We propose a new simulator, training approach, and policy architecture, collectively called SOUS VIDE, for end-to-end visual drone navigation. Our trained policies exhibit zero-shot sim-to-real transfer with robust real-world performance using only onboard perception and computation. Our simulator, called FiGS, couples a computationally simple drone dynamics model with a high visual fidelity Gaussian Splatting scene reconstruction. FiGS can quickly simulate drone flights producing photorealistic images at up to 130 fps. We use FiGS to collect 100k-300k image/state-action pairs from an expert MPC with privileged state and dynamics information, randomized over dynamics parameters and spatial disturbances. We then distill this expert MPC into an end-to-end visuomotor policy with a lightweight neural architecture, called SV-Net. SV-Net processes color image, optical flow and IMU data streams into low-level thrust and body rate commands at 20 Hz onboard a drone. Crucially, SV-Net includes a learned module for low-level control that adapts at runtime to variations in drone dynamics. In a campaign of 105 hardware experiments, we show SOUS VIDE policies to be robust to 30% mass variations, 40 m/s wind gusts, 60% changes in ambient brightness, shifting or removing objects from the scene, and people moving aggressively through the drone's visual field. Code, data, and experiment videos can be found on our project page: https://stanfordmsl.github.io/SousVide/.