# Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning

arXiv: https://arxiv.org/abs/1805.04969 · HTML: https://mlsanity.com/paper/1805.04969/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 13 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://mlsanity.com/paper/1506.07285/) (2015)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection](https://mlsanity.com/paper/1702.05552/) (2017)
- [Real time Detection of Lane Markers in Urban Streets](https://mlsanity.com/paper/1411.7113/) (2014)
- [Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behavior](https://mlsanity.com/paper/1704.05566/) (2017)
- [InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations](https://mlsanity.com/paper/1703.08840/) (2017)
- [Tracking by Prediction: A Deep Generative Model for Mutli-Person localisation and Tracking](https://mlsanity.com/paper/1803.03347/) (2018)
- [Tree Memory Networks for Modelling Long-term Temporal Dependencies](https://mlsanity.com/paper/1703.04706/) (2017)
- [Task Specific Visual Saliency Prediction with Memory Augmented Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1803.03354/) (2018)

## Built on by (most cited first, 3 of 3)

- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [World Models with Hints of Large Language Models for Goal Achieving](https://mlsanity.com/paper/2406.07381/) (2024)
- [BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay](https://mlsanity.com/paper/2402.14194/) (2024)

## Abstract (arXiv)

This paper presents a novel framework for automatic learning of complex strategies in human decision making. The task that we are interested in is to better facilitate long term planning for complex, multi-step events. We observe temporal relationships at the subtask level of expert demonstrations, and determine the different strategies employed in order to successfully complete a task. To capture the relationship between the subtasks and the overall goal, we utilise two external memory modules, one for capturing dependencies within a single expert demonstration, such as the sequential relationship among different sub tasks, and a global memory module for modelling task level characteristics such as best practice employed by different humans based on their domain expertise. Furthermore, we demonstrate how the hidden state representation of the memory can be used as a reward signal to smooth the state transitions, eradicating subtle changes. We evaluate the effectiveness of the proposed model for an autonomous highway driving application, where we demonstrate its capability to learn different expert policies and outperform state-of-the-art methods. The scope in industrial applications extends to any robotics and automation application which requires learning from complex demonstrations containing series of subtasks.