mlsanityData to 7 Oct 2026

Driving from Vision through Differentiable Optimal Control

arXiv 2403.15102 ↗First posted 22 Mar 2024cs.ROcited by 2

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

This paper proposes DriViDOC: a framework for Driving from Vision through Differentiable Optimal Control, and its application to learn autonomous driving controllers from human demonstrations. DriViDOC combines the automatic inference of relevant features from camera frames with the properties of nonlinear model predictive control (NMPC), such as constraint satisfaction. Our approach leverages the differentiability of parametric NMPC, allowing for end-to-end learning of the driving model from images to control. The model is trained on an offline dataset comprising various human demonstrations collected on a motion-base driving simulator. During online testing, the model demonstrates successful imitation of different driving styles, and the interpreted NMPC parameters provide insights into the achievement of specific driving behaviors. Our experimental results show that DriViDOC outperforms other methods involving NMPC and neural networks, exhibiting an average improvement of 20% in imitation scores.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 14 of its 26 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

14 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Evaluation of MPC-based Imitation Learning for Human-like Autonomous Driving22 Nov

Other references · 13

  1. 2016End to End Learning for Self-Driving Carscited by 1,000
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  4. 2020A Survey of End-to-End Driving: Architectures and Training Methodscited by 123
  5. 2019Data-driven Economic NMPC using Reinforcement Learningcited by 55
  6. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehiclescited by 36
  7. 2022PyPose: A Library for Robot Learning with Physics-based Optimizationcited by 29
  8. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 27
  9. 2021A predictive safety filter for learning-based racing controlcited by 17
  10. 2022A Hierarchical Approach for Strategic Motion Planning in Autonomous Racingcited by 5
  11. 2023Experimental Validation of Safe MPC for Autonomous Driving in Uncertain Environmentscited by 3
  12. 2024Real-time MPC with Control Barrier Functions for Autonomous Driving using Safety Enhanced Collocationcited by 3
1 more
  1. 2021Real-time Nonlinear MPC Strategy with Full Vehicle Validation for Autonomous Drivingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Imitation Learning from Observations: An Autoregressive Mixture of Experts Approachcited by 0
  2. 2026lapanda: A Matrix-Free Differentiable Solver for Nonconvex Constrained Optimization Layerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
  2. 2022MPC-based Imitation Learning for Safe and Human-like Autonomous Driving
  3. 2019Learning a Decision Module by Imitating Driver's Control Behaviors
  4. 2023Multi-Abstractive Neural Controller: An Efficient Hierarchical Control Architecture for Interactive Driving
  5. 2019Urban Driving with Conditional Imitation Learning
  6. 2024Towards Human-Like Driving: Active Inference in Autonomous Vehicle Control
  7. 2021How To Not Drive: Learning Driving Constraints from Demonstration
  8. 2025CARIL: Confidence-Aware Regression in Imitation Learning for Autonomous Driving