mlsanityData to 7 Oct 2026

A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo

arXiv 2406.12563 ↗First posted 18 Jun 2024cs.LG cs.CV cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

Racing autonomous cars faster than the best human drivers has been a longstanding grand challenge for the fields of Artificial Intelligence and robotics. Recently, an end-to-end deep reinforcement learning agent met this challenge in a high-fidelity racing simulator, Gran Turismo. However, this agent relied on global features that require instrumentation external to the car. This paper introduces, to the best of our knowledge, the first super-human car racing agent whose sensor input is purely local to the car, namely pixels from an ego-centric camera view and quantities that can be sensed from on-board the car, such as the car's velocity. By leveraging global features only at training time, the learned agent is able to outperform the best human drivers in time trial (one car on the track at a time) races using only local input features. The resulting agent is evaluated in Gran Turismo 7 on multiple tracks and cars. Detailed ablation experiments demonstrate the agent's strong reliance on visual inputs, making it the first vision-based super-human car racing agent.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

31 of 54 references.

References · 31

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2018YOLOv3: An Incremental Improvementcited by 2,652
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2018World Modelscited by 1,444
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  12. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
19 more
  1. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  2. 2020Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learningcited by 60
  3. 2022Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racingcited by 56
  4. 2018End-to-End Race Driving with Deep Reinforcement Learningcited by 38
  5. 2018Deep Learning under Privileged Information Using Heteroscedastic Dropoutcited by 26
  6. 2019Stochastic Dynamic Games in Belief Spacecited by 23
  7. 2021Learn-to-Race: A Multimodal Control Environment for Autonomous Racingcited by 23
  8. 2021Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learningcited by 21
  9. 2017A Non-Cooperative Game Approach to Autonomous Racingcited by 18
  10. 2021Robust Model Predictive Path Integral Control: Analysis and Performance Guaranteescited by 17
  11. 2019F1/10: An Open-Source Autonomous Cyber-Physical Platformcited by 13
  12. 2020Optimization-Based Hierarchical Motion Planning for Autonomous Racingcited by 11
  13. 2020Accurate, Low-Latency Visual Perception for Autonomous Racing:Challenges, Mechanisms, and Practical Solutionscited by 11
  14. 2020FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesiscited by 10
  15. 2021Formula RL: Deep Reinforcement Learning for Autonomous Racing using Telemetry Datacited by 10
  16. 2020Privileged Information Dropout in Reinforcement Learningcited by 8
  17. 2021Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Predictioncited by 7
  18. 2020Minimum Race-Time Planning-Strategy for an Autonomous Electric Racecarcited by 4
  19. 2021Expert Human-Level Driving in Gran Turismo Sport Using Deep Reinforcement Learning with Image-based Representationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 3.

12 papers.

  1. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  2. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  3. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 5
  4. 2025A Champion-level Vision-based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7cited by 4
  5. 2025Optimistic critics can empower small actorscited by 4
  6. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  7. 2025Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Accesscited by 2
  8. 2025Real-World Reinforcement Learning of Active Perception Behaviorscited by 1
  9. 2025Context-Aware Model-Based Reinforcement Learning for Autonomous Racingcited by 0
  10. 2026Learning to Play Blackjack: A Curriculum Learning Perspectivecited by 0
  11. 2026Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulationcited by 0
  12. 2026Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidancecited by 0