# BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay

arXiv: https://arxiv.org/abs/2402.14194 · HTML: https://mlsanity.com/paper/2402.14194/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing](https://mlsanity.com/paper/2202.07008/) (2022)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information](https://mlsanity.com/paper/1810.01266/) (2018)
- [ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning](https://mlsanity.com/paper/2309.05915/) (2023)
- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [A Taxonomy and Review of Algorithms for Modeling and Predicting Human Driver Behavior](https://mlsanity.com/paper/2006.08832/) (2020)
- [How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvement](https://mlsanity.com/paper/2303.02073/) (2023)
- [Residual Policy Learning for Vehicle Control of Autonomous Racing Cars](https://mlsanity.com/paper/2302.07035/) (2023)
- [A Gaussian Process Model for Opponent Prediction in Autonomous Racing](https://mlsanity.com/paper/2204.12533/) (2022)
- [Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1805.04969/) (2018)
- [Task Phasing: Automated Curriculum Learning from Demonstrations](https://mlsanity.com/paper/2210.10999/) (2022)

## Built on by (most cited first, 5 of 5)

- [Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing](https://mlsanity.com/paper/2410.10062/) (2024)
- [Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulation](https://mlsanity.com/paper/2512.05812/) (2025)
- [Structured Imitation Learning of Interactive Policies through Inverse Games](https://mlsanity.com/paper/2511.12848/) (2025)
- [Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints](https://mlsanity.com/paper/2603.05842/) (2026)
- [Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control](https://mlsanity.com/paper/2604.03023/) (2026)

## Abstract (arXiv)

Imitation learning learns a policy from demonstrations without requiring hand-designed reward functions. In many robotic tasks, such as autonomous racing, imitated policies must model complex environment dynamics and human decision-making. Sequence modeling is highly effective in capturing intricate patterns of motion sequences but struggles to adapt to new environments or distribution shifts that are common in real-world robotics tasks. In contrast, Adversarial Imitation Learning (AIL) can mitigate this effect, but struggles with sample inefficiency and handling complex motion patterns. Thus, we propose BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a Behavior Transformer (BeT) policy from human demonstrations with online AIL. BeTAIL adds an AIL residual policy to the BeT policy to model the sequential decision-making process of human experts and correct for out-of-distribution states or shifts in environment dynamics. We test BeTAIL on three challenges with expert-level demonstrations of real human gameplay in Gran Turismo Sport. Our proposed residual BeTAIL reduces environment interactions and improves racing performance and stability, even when the BeT is pretrained on different tracks than downstream learning. Videos and code available at: https://sites.google.com/berkeley.edu/BeTAIL/home.