mlsanityData to 7 Oct 2026

Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving

arXiv 2203.02401 ↗First posted 4 Mar 2022cs.RO cs.CV cs.LGcited by 10

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

Guaranteeing safety of perception-based learning systems is challenging due to the absence of ground-truth state information unlike in state-aware control scenarios. In this paper, we introduce a safety guaranteed learning framework for vision-based end-to-end autonomous driving. To this end, we design a learning system equipped with differentiable control barrier functions (dCBFs) that is trained end-to-end by gradient descent. Our models are composed of conventional neural network architectures and dCBFs. They are interpretable at scale, achieve great test performance under limited training data, and are safety guaranteed in a series of autonomous driving scenarios such as lane keeping and obstacle avoidance. We evaluated our framework in a sim-to-real environment, and tested on a real autonomous car, achieving safe lane following and obstacle avoidance via Augmented Reality (AR) and real parked vehicles.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 36 of its 61 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

36 of 61 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016End to End Learning for Self-Driving Carscited by 1,000
  4. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  5. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  6. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  7. 2020Liquid Time-constant Networkscited by 160
  8. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  9. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  10. 2020Learning Control Barrier Functions from Expert Demonstrationscited by 88
  11. 2019Urban Driving with Conditional Imitation Learningcited by 84
  12. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 82
24 more
  1. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  2. 2019Learning for Safety-Critical Control with Control Barrier Functionscited by 66
  3. 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 61
  4. 2019AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithmscited by 39
  5. 2020Neural Certificates for Safe Control Policiescited by 35
  6. 2018Variational End-to-End Navigation and Localizationcited by 34
  7. 2017Safe Learning of Quadrotor Dynamics Using Barrier Certificatescited by 32
  8. 2019Control Barrier Functions for Systems with High Relative Degreecited by 30
  9. 2020On The Verification of Neural ODEs with Stochastic Guaranteescited by 24
  10. 2020Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEscited by 22
  11. 2021Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racingcited by 22
  12. 2020ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllerscited by 17
  13. 2021Learning Interactive Driving Policies via Data-driven Simulationcited by 17
  14. 2020Learning Safe Neural Network Controllers with Barrier Certificatescited by 16
  15. 2020The NVIDIA PilotNet Experimentscited by 15
  16. 2018End to End Vehicle Lateral Control Using a Single Fisheye Cameracited by 13
  17. 2021GoTube: Scalable Stochastic Verification of Continuous-Depth Modelscited by 12
  18. 2021BarrierNet: A Safety-Guaranteed Layer for Neural Networkscited by 11
  19. 2020Training Neural Network Controllers Using Control Barrier Functions in the Presence of Disturbancescited by 9
  20. 2017Sensor Fusion for Robot Control through Deep Reinforcement Learningcited by 7
  21. 2021Recursive Feasibility Guided Optimal Parameter Adaptation of Differential Convex Optimization Policies for Safety-Critical Systemscited by 7
  22. 2020Sufficient Conditions for Feasibility of Optimal Control Problems Using Control Barrier Functionscited by 6
  23. 2019Self-triggered Control for Safety Critical Systems using Control Barrier Functionscited by 5
  24. 2021Comparison between safety methods control barrier function vs. reachability analysiscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2; 2024: 2; 2025: 2.

10 papers.

  1. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  2. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  3. 2022On the Forward Invariance of Neural ODEscited by 10
  4. 2022Differentiable Constrained Imitation Learning for Robot Motion Planning and Controlcited by 7
  5. 2022Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gapcited by 5
  6. 2025Learning Conservative Neural Control Barrier Functions from Offline Datacited by 4
  7. 2025Conformal Safety Shielding for Imperfect-Perception Agentscited by 3
  8. 2024Learning Ensembles of Vision-based Safety Control Filterscited by 2
  9. 2022Interpreting Neural Policies with Disentangled Tree Representationscited by 1
  10. 2024Enhanced Safety in Autonomous Driving: Integrating Latent State Diffusion Model for End-to-End Navigationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Driving
  2. 2024Integrating Decision-Making Into Differentiable Optimization Guided Learning for End-to-End Planning of Autonomous Vehicles
  3. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Review
  4. 2023Combining Policy Gradient and Safety-Based Control for Autonomous Driving
  5. 2025Drive in Corridors: Enhancing the Safety of End-to-end Autonomous Driving via Corridor Learning and Planning
  6. 2023Learning Stability Attention in Vision-based End-to-end Driving Policies
  7. 2026Data-Driven Risk Fields for Safer End-to-End Autonomous Driving
  8. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey