# Controlling Steering with Energy-Based Models

arXiv: https://arxiv.org/abs/2301.12264 · HTML: https://mlsanity.com/paper/2301.12264/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 28 Jan 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Learning to Drive from Simulation without Real World Labels](https://mlsanity.com/paper/1812.03823/) (2018)
- [Autonomy 2.0: Why is self-driving always 5 years away?](https://mlsanity.com/paper/2107.08142/) (2021)
- [End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies](https://mlsanity.com/paper/1710.03804/) (2017)
- [The NVIDIA PilotNet Experiments](https://mlsanity.com/paper/2010.08776/) (2020)
- [Reimagining an autonomous vehicle](https://mlsanity.com/paper/2108.05805/) (2021)
- [Learning Accurate and Human-Like Driving using Semantic Maps and Attention](https://mlsanity.com/paper/2007.07218/) (2020)
- [LiDAR-as-Camera for End-to-End Driving](https://mlsanity.com/paper/2206.15170/) (2022)
- [Two-stream convolutional networks for end-to-end learning of self-driving cars](https://mlsanity.com/paper/1811.05785/) (2018)

## Built on by (most cited first, 2 of 2)

- [Combating the effects of speed and delays in end-to-end self-driving](https://mlsanity.com/paper/2312.06670/) (2023)
- [Exploring multimodal implicit behavior learning for vehicle navigation in simulated cities](https://mlsanity.com/paper/2509.15400/) (2025)

## Abstract (arXiv)

So-called implicit behavioral cloning with energy-based models has shown promising results in robotic manipulation tasks. We tested if the method's advantages carry on to controlling the steering of a real self-driving car with an end-to-end driving model. We performed an extensive comparison of the implicit behavioral cloning approach with explicit baseline approaches, all sharing the same neural network backbone architecture. Baseline explicit models were trained with regression (MAE) loss, classification loss (softmax and cross-entropy on a discretization), or as mixture density networks (MDN). While models using the energy-based formulation performed comparably to baseline approaches in terms of safety driver interventions, they had a higher whiteness measure, indicating higher jerk. To alleviate this, we show two methods that can be used to improve the smoothness of steering. We confirmed that energy-based models handle multimodalities slightly better than simple regression, but this did not translate to significantly better driving ability. We argue that the steering-only road-following task has too few multimodalities to benefit from energy-based models. This shows that applying implicit behavioral cloning to real-world tasks can be challenging, and further investigation is needed to bring out the theoretical advantages of energy-based models.