# Robust Guarantees for Perception-Based Control

arXiv: https://arxiv.org/abs/1907.03680 · HTML: https://mlsanity.com/paper/1907.03680/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 20 of 20)

- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations](https://mlsanity.com/paper/2111.09971/) (2021)
- [Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory](https://mlsanity.com/paper/2206.06553/) (2022)
- [Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction](https://mlsanity.com/paper/2304.00194/) (2023)
- [Learning Linear Dynamics from Bilinear Observations](https://mlsanity.com/paper/2409.16499/) (2024)
- [System-level Safety Guard: Safe Tracking Control through Uncertain Neural Network Dynamics Models](https://mlsanity.com/paper/2312.06810/) (2023)
- [Autonomous Driving With Perception Uncertainties: Deep-Ensemble Based Adaptive Cruise Control](https://mlsanity.com/paper/2403.15577/) (2024)
- [Efficient Motion Planning for Manipulators with Control Barrier Function-Induced Neural Controller](https://mlsanity.com/paper/2404.01184/) (2024)
- [Conformal Safety Shielding for Imperfect-Perception Agents](https://mlsanity.com/paper/2506.17275/) (2025)
- [Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning](https://mlsanity.com/paper/2402.09290/) (2024)
- [Multi-task UNet architecture for end-to-end autonomous driving](https://mlsanity.com/paper/2112.08967/) (2021)
- [Refining Perception Contracts: Case Studies in Vision-based Safe Auto-landing](https://mlsanity.com/paper/2311.08652/) (2023)
- [Learning to Control an Unstable System with One Minute of Data: Leveraging Gaussian Process Differentiation in Predictive Control](https://mlsanity.com/paper/2103.04548/) (2021)
- [Learning Lipschitz Feedback Policies from Expert Demonstrations: Closed-Loop Guarantees, Generalization and Robustness](https://mlsanity.com/paper/2103.16629/) (2021)
- [Certifying Safety in Reinforcement Learning under Adversarial Perturbation Attacks](https://mlsanity.com/paper/2212.14115/) (2022)
- [Symbolic Perception Risk in Autonomous Driving](https://mlsanity.com/paper/2303.09416/) (2023)
- [Robust Model Predictive Control Design for Autonomous Vehicles with Perception-based Observers](https://mlsanity.com/paper/2509.05201/) (2025)
- [VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving](https://mlsanity.com/paper/2607.05180/) (2026)
- [Correcting Learning-based Perception for Safety](https://mlsanity.com/paper/2609.22108/) (2026)

## Abstract (arXiv)

Motivated by vision-based control of autonomous vehicles, we consider the problem of controlling a known linear dynamical system for which partial state information, such as vehicle position, is extracted from complex and nonlinear data, such as a camera image. Our approach is to use a learned perception map that predicts some linear function of the state and to design a corresponding safe set and robust controller for the closed loop system with this sensing scheme. We show that under suitable smoothness assumptions on both the perception map and the generative model relating state to complex and nonlinear data, parameters of the safe set can be learned via appropriately dense sampling of the state space. We then prove that the resulting perception-control loop has favorable generalization properties. We illustrate the usefulness of our approach on a synthetic example and on the self-driving car simulation platform CARLA.