Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Driving
On the map: End-to-end autonomous driving → VLMs for autonomous driving
Abstract (arXiv)
End-to-end vision-based autonomous driving has achieved impressive success, but safety remains a major concern. The safe control problem has been addressed in low-dimensional settings using safety filters, e.g., those based on control barrier functions. Designing safety filters for vision-based controllers in the high-dimensional settings of autonomous driving can similarly alleviate the safety problem, but is significantly more challenging. In this paper, we address this challenge by using frozen pre-trained vision representation models as perception backbones to design vision-based safety filters, inspired by these models' success as backbones of robotic control policies. We empirically evaluate the offline performance of four common pre-trained vision models in this context. We try three existing methods for training safety filters for black-box dynamics, as the dynamics over representation spaces are not known. We use the DeepAccident dataset that consists of action-annotated videos from multiple cameras on vehicles in CARLA simulating real accident scenarios. Our results show that the filters resulting from our approach are competitive with the ones that are given the ground truth state of the ego vehicle and its environment.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 35 of its 42 references that are in the map. First posted on arXiv 29 Oct 2024.
Builds on
35 of 42 references.
References · 35
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2017Constrained Policy Optimizationcited by 928
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
- 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
- 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
- 2019Control Barrier Functions: Theory and Applicationscited by 332
23 more
- 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 261
- 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
- 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
- 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
- 2023DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Drivingcited by 73
- 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
- 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
- 2021Robust Control Barrier-Value Functions for Safety-Critical Controlcited by 47
- 2020Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimizationcited by 34
- 2020Guaranteeing Safety of Learned Perception Modules via Measurement-Robust Control Barrier Functionscited by 32
- 2021Verification of Image-based Neural Network Controllers Using Generative Modelscited by 27
- 2021Verifying Controllers with Convolutional Neural Network-based Perception: A Case for Intelligible, Safe, and Precise Abstractionscited by 19
- 2022NNLander-VeriF: A Neural Network Formal Verification Framework for Vision-Based Autonomous Aircraft Landingcited by 19
- 2023Closed-loop Analysis of Vision-based Autonomous Systems: A Case Studycited by 18
- 2022SABLAS: Learning Safe Control for Black-box Dynamical Systemscited by 17
- 2022Refining Control Barrier Functions through Hamilton-Jacobi Reachabilitycited by 15
- 2022Enforcing safety for vision-based controllers via Control Barrier Functions and Neural Radiance Fieldscited by 15
- 2023In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution Statescited by 15
- 2018Permissive Barrier Certificates for Safe Stabilization Using Sum-of-squarescited by 14
- 2021Verification and Synthesis of Control Barrier Functionscited by 13
- 2022Control Barrier Functions and Input-to-State Safety with Application to Automated Vehiclescited by 11
- 2024Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanescited by 8
- 2024Scalable Surrogate Verification of Image-based Neural Network Control Systems using Composition and Unrollingcited by 8
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Designing Latent Safety Filters using Pre-Trained Vision Models
- 2022Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving
- 2025VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving
- 2024An Examination of Offline-Trained Encoders in Vision-Based Deep Reinforcement Learning for Autonomous Driving
- 2026FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder
- 2026VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
- 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Models
- 2025Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM