# Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory

arXiv: https://arxiv.org/abs/2206.06553 · HTML: https://mlsanity.com/paper/2206.06553/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach](https://mlsanity.com/paper/1801.10578/) (2018)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [Learning Certified Control using Contraction Metric](https://mlsanity.com/paper/2011.12569/) (2020)
- [Control Contraction Metrics: Convex and Intrinsic Criteria for Nonlinear Feedback Design](https://mlsanity.com/paper/1503.03144/) (2015)
- [Guaranteeing Safety of Learned Perception Modules via Measurement-Robust Control Barrier Functions](https://mlsanity.com/paper/2010.16001/) (2020)
- [Funnel Libraries for Real-Time Robust Feedback Motion Planning](https://mlsanity.com/paper/1601.04037/) (2016)
- [Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constants](https://mlsanity.com/paper/2010.08993/) (2020)
- [Learning Safe, Generalizable Perception-based Hybrid Control with Certificates](https://mlsanity.com/paper/2201.00932/) (2022)
- [Neural Contraction Metrics for Robust Estimation and Control: A Convex Optimization Approach](https://mlsanity.com/paper/2006.04361/) (2020)
- [How to Train Your Differentiable Filter](https://mlsanity.com/paper/2012.14313/) (2020)
- [Robust Guarantees for Perception-Based Control](https://mlsanity.com/paper/1907.03680/) (2019)
- [Probably Approximately Correct Vision-Based Planning using Motion Primitives](https://mlsanity.com/paper/2002.12852/) (2020)
- [Learning-based Robust Motion Planning with Guaranteed Stability: A Contraction Theory Approach](https://mlsanity.com/paper/2102.12668/) (2021)
- [Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems](https://mlsanity.com/paper/2104.08695/) (2021)
- [Measurement-Robust Control Barrier Functions: Certainty in Safety with Uncertainty in State](https://mlsanity.com/paper/2104.14030/) (2021)
- [Output-Feedback Control of Nonlinear Systems using Control Contraction Metrics and Convex Optimization](https://mlsanity.com/paper/1406.1256/) (2014)
- [Towards Integrated Perception and Motion Planning with Distributionally Robust Risk Constraints](https://mlsanity.com/paper/2002.02928/) (2020)
- [Contraction Analysis of Discrete-time Stochastic Systems](https://mlsanity.com/paper/2106.05635/) (2021)
- [Robust Sample-Based Output-Feedback Path Planning](https://mlsanity.com/paper/2105.14118/) (2021)

## Built on by (most cited first, 19 of 19)

- [Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching](https://mlsanity.com/paper/2306.14079/) (2023)
- [Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction](https://mlsanity.com/paper/2304.00194/) (2023)
- [Statistical Safety and Robustness Guarantees for Feedback Motion Planning of Unknown Underactuated Stochastic Systems](https://mlsanity.com/paper/2212.06874/) (2022)
- [Synthesizing Stable Reduced-Order Visuomotor Policies for Nonlinear Systems via Sums-of-Squares Optimization](https://mlsanity.com/paper/2304.12405/) (2023)
- [Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning](https://mlsanity.com/paper/2311.06481/) (2023)
- [Safety Beyond the Training Data: Robust Out-of-Distribution MPC via Conformalized System Level Synthesis](https://mlsanity.com/paper/2602.12047/) (2026)
- [Safe Large-Scale Robust Nonlinear MPC in Milliseconds via Reachability-Constrained System Level Synthesis on the GPU](https://mlsanity.com/paper/2604.07644/) (2026)
- [Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions](https://mlsanity.com/paper/2412.13157/) (2024)
- [Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC](https://mlsanity.com/paper/2606.15594/) (2026)
- [VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis](https://mlsanity.com/paper/2604.24894/) (2026)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Formal Safety Verification and Refinement for Generative Motion Planners via Certified Local Stabilization](https://mlsanity.com/paper/2509.19688/) (2025)
- [Parallel Differentiable Reachability for Learning and Planning with Certified Neural Dynamics and Controllers](https://mlsanity.com/paper/2605.25346/) (2026)
- [Scalable Data-Driven Reachability Analysis and Control via Koopman Operators with Conformal Coverage Guarantees](https://mlsanity.com/paper/2601.01076/) (2026)
- [Anytime Probabilistically Constrained Provably Convergent Online Belief Space Planning](https://mlsanity.com/paper/2411.06711/) (2024)
- [Verification of Visual Controllers via Compositional Geometric Transformations](https://mlsanity.com/paper/2507.04523/) (2025)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [Correcting Learning-based Perception for Safety](https://mlsanity.com/paper/2609.22108/) (2026)
- [Latent Safety Filters: When a Lossy Encoder Admits a Transferable Certificate](https://mlsanity.com/paper/2610.04297/) (2026)

## Abstract (arXiv)

We present a motion planning algorithm for a class of uncertain control-affine nonlinear systems which guarantees runtime safety and goal reachability when using high-dimensional sensor measurements (e.g., RGB-D images) and a learned perception module in the feedback control loop. First, given a dataset of states and observations, we train a perception system that seeks to invert a subset of the state from an observation, and estimate an upper bound on the perception error which is valid with high probability in a trusted domain near the data. Next, we use contraction theory to design a stabilizing state feedback controller and a convergent dynamic state observer which uses the learned perception system to update its state estimate. We derive a bound on the trajectory tracking error when this controller is subjected to errors in the dynamics and incorrect state estimates. Finally, we integrate this bound into a sampling-based motion planner, guiding it to return trajectories that can be safely tracked at runtime using sensor data. We demonstrate our approach in simulation on a 4D car, a 6D planar quadrotor, and a 17D manipulation task with RGB(-D) sensor measurements, demonstrating that our method safely and reliably steers the system to the goal, while baselines that fail to consider the trusted domain or state estimation errors can be unsafe.