# Learning Safe, Generalizable Perception-based Hybrid Control with Certificates

arXiv: https://arxiv.org/abs/2201.00932 · HTML: https://mlsanity.com/paper/2201.00932/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2022.

## Built on by (most cited first, 24 of 24)

- [Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory](https://mlsanity.com/paper/2206.06553/) (2022)
- [Closed-loop Analysis of Vision-based Autonomous Systems: A Case Study](https://mlsanity.com/paper/2302.04634/) (2023)
- [Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction](https://mlsanity.com/paper/2304.00194/) (2023)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specifications](https://mlsanity.com/paper/2201.12231/) (2022)
- [Joint Differentiable Optimization and Verification for Certified Reinforcement Learning](https://mlsanity.com/paper/2201.12243/) (2022)
- [State-Wise Safe Reinforcement Learning With Pixel Observations](https://mlsanity.com/paper/2311.02227/) (2023)
- [Synthesizing Stable Reduced-Order Visuomotor Policies for Nonlinear Systems via Sums-of-Squares Optimization](https://mlsanity.com/paper/2304.12405/) (2023)
- [Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory](https://mlsanity.com/paper/2306.04026/) (2023)
- [Two-Stage Learning of Stabilizing Neural Controllers via Zubov Sampling and Iterative Domain Expansion](https://mlsanity.com/paper/2506.01356/) (2025)
- [Efficient Motion Planning for Manipulators with Control Barrier Function-Induced Neural Controller](https://mlsanity.com/paper/2404.01184/) (2024)
- [Logically Constrained Robotics Transformers for Enhanced Perception-Action Planning](https://mlsanity.com/paper/2408.05336/) (2024)
- [ConBaT: Control Barrier Transformer for Safe Policy Learning](https://mlsanity.com/paper/2303.04212/) (2023)
- [Conformal Safety Shielding for Imperfect-Perception Agents](https://mlsanity.com/paper/2506.17275/) (2025)
- [Output Feedback Tube MPC-Guided Data Augmentation for Robust, Efficient Sensorimotor Policy Learning](https://mlsanity.com/paper/2210.10127/) (2022)
- [Learning Ensembles of Vision-based Safety Control Filters](https://mlsanity.com/paper/2412.02029/) (2024)
- [T-CBF: Traversability-based Control Barrier Function to Navigate Vertically Challenging Terrain](https://mlsanity.com/paper/2503.06083/) (2025)
- [ORN-CBF: Learning Observation-conditioned Residual Neural Control Barrier Functions via Hypernetworks](https://mlsanity.com/paper/2509.16614/) (2025)
- [Assumption Generation for the Verification of Learning-Enabled Autonomous Systems](https://mlsanity.com/paper/2305.18372/) (2023)
- [Online Efficient Safety-Critical Control for Mobile Robots in Unknown Dynamic Multi-Obstacle Environments](https://mlsanity.com/paper/2402.16449/) (2024)
- [Fault Tolerant Neural Control Barrier Functions for Robotic Systems under Sensor Faults and Attacks](https://mlsanity.com/paper/2402.18677/) (2024)
- [Learned Lyapunov Shielding for Adaptive Control](https://mlsanity.com/paper/2605.06934/) (2026)
- [ShardNet: Training Neural Controllers with Hard, Non-Convex Constraints](https://mlsanity.com/paper/2606.30935/) (2026)
- [Correcting Learning-based Perception for Safety](https://mlsanity.com/paper/2609.22108/) (2026)

## Abstract (arXiv)

Many robotic tasks require high-dimensional sensors such as cameras and Lidar to navigate complex environments, but developing certifiably safe feedback controllers around these sensors remains a challenging open problem, particularly when learning is involved. Previous works have proved the safety of perception-feedback controllers by separating the perception and control subsystems and making strong assumptions on the abilities of the perception subsystem. In this work, we introduce a novel learning-enabled perception-feedback hybrid controller, where we use Control Barrier Functions (CBFs) and Control Lyapunov Functions (CLFs) to show the safety and liveness of a full-stack perception-feedback controller. We use neural networks to learn a CBF and CLF for the full-stack system directly in the observation space of the robot, without the need to assume a separate perception-based state estimator. Our hybrid controller, called LOCUS (Learning-enabled Observation-feedback Control Using Switching), can safely navigate unknown environments, consistently reach its goal, and generalizes safely to environments outside of the training dataset. We demonstrate LOCUS in experiments both in simulation and in hardware, where it successfully navigates a changing environment using feedback from a Lidar sensor.