mlsanityData to 7 Oct 2026

Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Cars

arXiv 2206.10249 ↗First posted 21 Jun 2022cs.HC cs.CL cs.CVcited by 2

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

This paper presents a novel approach that supports natural language voice instructions to guide deep reinforcement learning (DRL) algorithms when training self-driving cars. DRL methods are popular approaches for autonomous vehicle (AV) agents. However, most existing methods are sample- and time-inefficient and lack a natural communication channel with the human expert. In this paper, how new human drivers learn from human coaches motivates us to study new ways of human-in-the-loop learning and a more natural and approachable training interface for the agents. We propose incorporating natural language voice instructions (NLI) in model-based deep reinforcement learning to train self-driving cars. We evaluate the proposed method together with a few state-of-the-art DRL methods in the CARLA simulator. The results show that NLI can help ease the training process and significantly boost the agents' learning speed.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 17 of its 32 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

17 of 32 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  9. 2019Learning by Cheatingcited by 361
  10. 2018Learning to Drive in a Daycited by 266
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  12. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
5 more
  1. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  2. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  3. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  4. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  5. 2018End-to-End Race Driving with Deep Reinforcement Learningcited by 38

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  2. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2021Modified DDPG car-following model with a real-world human driving experience with CARLA simulator
  2. 2024Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Driving
  3. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
  4. 2024Self-Driving Car Racing: Application of Deep Reinforcement Learning
  5. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving
  6. 2019Conditional Driving from Natural Language Instructions
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Survey
  8. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning