# Driving from Vision through Differentiable Optimal Control

arXiv: https://arxiv.org/abs/2403.15102 · HTML: https://mlsanity.com/paper/2403.15102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)

## Builds on (most cited first, 14 of 14)

- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation](https://mlsanity.com/paper/2209.10780/) (2022)
- [PyPose: A Library for Robot Learning with Physics-based Optimization](https://mlsanity.com/paper/2209.15428/) (2022)
- [A predictive safety filter for learning-based racing control](https://mlsanity.com/paper/2102.11907/) (2021)
- [A Hierarchical Approach for Strategic Motion Planning in Autonomous Racing](https://mlsanity.com/paper/2212.01607/) (2022)
- [Evaluation of MPC-based Imitation Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2211.12111/) (2022)
- [Experimental Validation of Safe MPC for Autonomous Driving in Uncertain Environments](https://mlsanity.com/paper/2305.03312/) (2023)
- [Real-time MPC with Control Barrier Functions for Autonomous Driving using Safety Enhanced Collocation](https://mlsanity.com/paper/2401.06648/) (2024)
- [Real-time Nonlinear MPC Strategy with Full Vehicle Validation for Autonomous Driving](https://mlsanity.com/paper/2110.03349/) (2021)

## Built on by (most cited first, 2 of 2)

- [Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach](https://mlsanity.com/paper/2411.08232/) (2024)
- [lapanda: A Matrix-Free Differentiable Solver for Nonconvex Constrained Optimization Layers](https://mlsanity.com/paper/2609.33630/) (2026)

## Abstract (arXiv)

This paper proposes DriViDOC: a framework for Driving from Vision through Differentiable Optimal Control, and its application to learn autonomous driving controllers from human demonstrations. DriViDOC combines the automatic inference of relevant features from camera frames with the properties of nonlinear model predictive control (NMPC), such as constraint satisfaction. Our approach leverages the differentiability of parametric NMPC, allowing for end-to-end learning of the driving model from images to control. The model is trained on an offline dataset comprising various human demonstrations collected on a motion-base driving simulator. During online testing, the model demonstrates successful imitation of different driving styles, and the interpreted NMPC parameters provide insights into the achievement of specific driving behaviors. Our experimental results show that DriViDOC outperforms other methods involving NMPC and neural networks, exhibiting an average improvement of 20% in imitation scores.