# Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction

arXiv: https://arxiv.org/abs/2304.00194 · HTML: https://mlsanity.com/paper/2304.00194/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Safe Planning in Dynamic Environments using Conformal Prediction](https://mlsanity.com/paper/2210.10254/) (2022)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [Adaptive Conformal Prediction for Motion Planning among Dynamic Agents](https://mlsanity.com/paper/2212.00278/) (2022)
- [Guaranteeing Safety of Learned Perception Modules via Measurement-Robust Control Barrier Functions](https://mlsanity.com/paper/2010.16001/) (2020)
- [Learning Safe, Generalizable Perception-based Hybrid Control with Certificates](https://mlsanity.com/paper/2201.00932/) (2022)
- [PAC-Bayes Control: Learning Policies that Provably Generalize to Novel Environments](https://mlsanity.com/paper/1806.04225/) (2018)
- [Robust Guarantees for Perception-Based Control](https://mlsanity.com/paper/1907.03680/) (2019)
- [Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations](https://mlsanity.com/paper/2111.09971/) (2021)
- [Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory](https://mlsanity.com/paper/2206.06553/) (2022)
- [Case Study: Verifying the Safety of an Autonomous Racing Car with a Neural Network Controller](https://mlsanity.com/paper/1910.11309/) (2019)
- [Probably Approximately Correct Vision-Based Planning using Motion Primitives](https://mlsanity.com/paper/2002.12852/) (2020)
- [Control Barrier Functions for Stochastic Systems](https://mlsanity.com/paper/2003.03498/) (2020)
- [Perception-Based Temporal Logic Planning in Uncertain Semantic Maps](https://mlsanity.com/paper/2012.10490/) (2020)
- [A Benchmark Comparison of Imitation Learning-based Control Policies for Autonomous Racing](https://mlsanity.com/paper/2209.15073/) (2022)
- [Measurement-Robust Control Barrier Functions: Certainty in Safety with Uncertainty in State](https://mlsanity.com/paper/2104.14030/) (2021)
- [Differentiable Safe Controller Design through Control Barrier Functions](https://mlsanity.com/paper/2209.10034/) (2022)
- [Neural Predictive Monitoring under Partial Observability](https://mlsanity.com/paper/2108.07134/) (2021)
- [Self-Supervised Online Learning for Safety-Critical Control using Stereo Vision](https://mlsanity.com/paper/2203.01404/) (2022)
- [Control Barrier Function Meets Interval Analysis: Safety-Critical Control with Measurement and Actuation Uncertainties](https://mlsanity.com/paper/2110.00915/) (2021)
- [Safe and Robust Observer-Controller Synthesis using Control Barrier Functions](https://mlsanity.com/paper/2211.14364/) (2022)
- [Feedback from Pixels: Output Regulation via Learning-Based Scene View Synthesis](https://mlsanity.com/paper/2103.10888/) (2021)
- [Observer-based Control Barrier Functions for Safety Critical Systems](https://mlsanity.com/paper/2110.00923/) (2021)
- [Online Optimization of Dynamical Systems with Deep Learning Perception](https://mlsanity.com/paper/2205.09574/) (2022)
- [Attacks on Perception-Based Control Systems: Modeling and Fundamental Limits](https://mlsanity.com/paper/2206.07150/) (2022)
- [Learning Certifiably Robust Controllers Using Fragile Perception](https://mlsanity.com/paper/2209.11328/) (2022)

## Built on by (most cited first, 16 of 16)

- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning](https://mlsanity.com/paper/2407.17460/) (2024)
- [How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy](https://mlsanity.com/paper/2308.12252/) (2023)
- [Towards Generalizable Safety in Crowd Navigation via Conformal Uncertainty Handling](https://mlsanity.com/paper/2508.05634/) (2025)
- [Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction](https://mlsanity.com/paper/2502.21186/) (2025)
- [Conformal Off-Policy Prediction for Multi-Agent Systems](https://mlsanity.com/paper/2403.16871/) (2024)
- [Conformal Safety Shielding for Imperfect-Perception Agents](https://mlsanity.com/paper/2506.17275/) (2025)
- [VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis](https://mlsanity.com/paper/2604.24894/) (2026)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion](https://mlsanity.com/paper/2506.12095/) (2025)
- [How Safe Will I Be Given What I Saw? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy](https://mlsanity.com/paper/2508.09346/) (2025)
- [Particle-Based Conformal Prediction for Contact-Aware Uncertainty Calibration in Stratified Configuration Spaces](https://mlsanity.com/paper/2608.09166/) (2026)
- [Statistical Contraction for Chance-Constrained Trajectory Optimization of Non-Gaussian Stochastic Systems](https://mlsanity.com/paper/2603.07092/) (2026)
- [Differentiable Optimization Layered Safety-Critical Control for Risk-Aware Navigation via Conformal Prediction](https://mlsanity.com/paper/2605.16327/) (2026)
- [A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification](https://mlsanity.com/paper/2608.29789/) (2026)

## Abstract (arXiv)

We consider perception-based control using state estimates that are obtained from high-dimensional sensor measurements via learning-enabled perception maps. However, these perception maps are not perfect and result in state estimation errors that can lead to unsafe system behavior. Stochastic sensor noise can make matters worse and result in estimation errors that follow unknown distributions. We propose a perception-based control framework that i) quantifies estimation uncertainty of perception maps, and ii) integrates these uncertainty representations into the control design. To do so, we use conformal prediction to compute valid state estimation regions, which are sets that contain the unknown state with high probability. We then devise a sampled-data controller for continuous-time systems based on the notion of measurement robust control barrier functions. Our controller uses idea from self-triggered control and enables us to avoid using stochastic calculus. Our framework is agnostic to the choice of the perception map, independent of the noise distribution, and to the best of our knowledge the first to provide probabilistic safety guarantees in such a setting. We demonstrate the effectiveness of our proposed perception-based controller for a LiDAR-enabled F1/10th car.