# Enforcing safety for vision-based controllers via Control Barrier Functions and Neural Radiance Fields

arXiv: https://arxiv.org/abs/2209.12266 · HTML: https://mlsanity.com/paper/2209.12266/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2022.

## Built on by (most cited first, 16 of 16)

- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates](https://mlsanity.com/paper/2405.14058/) (2024)
- [State-Wise Safe Reinforcement Learning With Pixel Observations](https://mlsanity.com/paper/2311.02227/) (2023)
- [How to Train Your Latent Control Barrier Function: Smooth Safety Filtering Under Hard-to-Model Constraints](https://mlsanity.com/paper/2511.18606/) (2025)
- [Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses](https://mlsanity.com/paper/2605.02900/) (2026)
- [System-level Safety Guard: Safe Tracking Control through Uncertain Neural Network Dynamics Models](https://mlsanity.com/paper/2312.06810/) (2023)
- [Designing Latent Safety Filters using Pre-Trained Vision Models](https://mlsanity.com/paper/2509.14758/) (2025)
- [Safe and Reliable Training of Learning-Based Aerospace Controllers](https://mlsanity.com/paper/2407.07088/) (2024)
- [Conformal Safety Shielding for Imperfect-Perception Agents](https://mlsanity.com/paper/2506.17275/) (2025)
- [Learning Ensembles of Vision-based Safety Control Filters](https://mlsanity.com/paper/2412.02029/) (2024)
- [Safety Certification in the Latent space using Control Barrier Functions and World Models](https://mlsanity.com/paper/2507.13871/) (2025)
- [Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Driving](https://mlsanity.com/paper/2410.22585/) (2024)
- [Learning Vision-Based Neural Network Controllers with Semi-Probabilistic Safety Guarantees](https://mlsanity.com/paper/2503.00191/) (2025)
- [Enhancing Feature Tracking Reliability for Visual Navigation using Real-Time Safety Filter](https://mlsanity.com/paper/2502.01092/) (2025)
- [Verification of Visual Controllers via Compositional Geometric Transformations](https://mlsanity.com/paper/2507.04523/) (2025)
- [Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation](https://mlsanity.com/paper/2609.36520/) (2026)

## Abstract (arXiv)

To navigate complex environments, robots must increasingly use high-dimensional visual feedback (e.g. images) for control. However, relying on high-dimensional image data to make control decisions raises important questions; particularly, how might we prove the safety of a visual-feedback controller? Control barrier functions (CBFs) are powerful tools for certifying the safety of feedback controllers in the state-feedback setting, but CBFs have traditionally been poorly-suited to visual feedback control due to the need to predict future observations in order to evaluate the barrier function. In this work, we solve this issue by leveraging recent advances in neural radiance fields (NeRFs), which learn implicit representations of 3D scenes and can render images from previously-unseen camera perspectives, to provide single-step visual foresight for a CBF-based controller. This novel combination is able to filter out unsafe actions and intervene to preserve safety. We demonstrate the effect of our controller in real-time simulation experiments where it successfully prevents the robot from taking dangerous actions.