mlsanityData to 7 Oct 2026

Reward (Mis)design for Autonomous Driving

arXiv 2104.13906 ↗First posted 28 Apr 2021cs.LGcited by 102

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

This article considers the problem of diagnosing certain common errors in reward design. Its insights are also applicable to the design of cost functions and performance metrics more generally. To diagnose common errors, we develop 8 simple sanity checks for identifying flaws in reward functions. These sanity checks are applied to reward functions from past work on reinforcement learning (RL) for autonomous driving (AD), revealing near-universal flaws in reward design for AD that might also exist pervasively across reward design for other tasks. Lastly, we explore promising directions that may aid the design of reward functions for AD in subsequent research, following a process of inquiry that can be adapted to other domains.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 28 of its 63 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

28 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learning6 Jan

Other references · 27

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  8. 2018Learning to Drive in a Daycited by 266
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Inverse Reward Designcited by 246
  11. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  12. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
15 more
  1. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  2. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  3. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
  4. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  5. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  6. 2018End-to-End Race Driving with Deep Reinforcement Learningcited by 38
  7. 2018Urban Driving with Multi-Objective Deep Reinforcement Learningcited by 30
  8. 2017Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environmentscited by 19
  9. 2020Towards Learning Multi-agent Negotiations via Self-Playcited by 17
  10. 2017Learning End-to-end Multimodal Sensor Policies for Autonomous Navigationcited by 16
  11. 2020Learning hierarchical behavior and motion planning for autonomous drivingcited by 15
  12. 2019LeTS-Drive: Driving in a Crowd by Learning from Tree Searchcited by 13
  13. 2019Dynamic Input for Deep Reinforcement Learning in Autonomous Drivingcited by 11
  14. 2019Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithmcited by 8
  15. 2019Quadratic Q-network for Learning Continuous Control for Autonomous Vehiclescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 14; 2024: 28; 2025: 26; 2026: 18.

102 papers.

Sort:

By area:

  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  4. 2023Reward Design with Language Modelscited by 257
  5. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  6. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  7. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  8. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  9. 2022Models of human preference for learning reward functionscited by 54
  10. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  11. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  12. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
38 more
  1. 2023Goodhart's Law in Reinforcement Learningcited by 37
  2. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  3. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  4. 2024HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Drivingcited by 31
  5. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  6. 2022Proximal Point Imitation Learningcited by 25
  7. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  8. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  9. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  10. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  11. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  12. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  13. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  14. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19
  15. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  16. 2024Human-compatible driving partners through data-regularized self-play reinforcement learningcited by 17
  17. 2024A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Drivingcited by 17
  18. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  19. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  20. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  21. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
  22. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  23. 2022Receding Horizon Inverse Reinforcement Learningcited by 10
  24. 2025Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learningcited by 10
  25. 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitionerscited by 10
  26. 2023Energy-based Potential Games for Joint Motion Forecasting and Controlcited by 9
  27. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  28. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Modelscited by 8
  29. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  30. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 8
  31. 2022Differentiable Constrained Imitation Learning for Robot Motion Planning and Controlcited by 7
  32. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  33. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Drivingcited by 7
  34. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  35. 2025A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulatorcited by 7
  36. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6
  37. 2026Learning to Drive in New Cities Without Human Demonstrationscited by 6
  38. 2023On a Connection between Differential Games, Optimal Control, and Energy-based Models for Multi-Agent Interactionscited by 4

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Checking for Reinforcement Learning in Autonomous Driving: One Can Do More Than You Think!
  2. 2023Risk-Aware Reward Shaping of Reinforcement Learning Agents for Autonomous Driving
  3. 2024LORD: Large Models based Opposite Reward Design for Autonomous Driving
  4. 2026DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving
  5. 2026Reward-Conditioned Attention: How Reward Design Shapes What Autonomous Driving Agents See
  6. 2024Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study
  7. 2022Driver Dojo: A Benchmark for Generalizable Reinforcement Learning for Autonomous Driving
  8. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures