mlsanityData to 7 Oct 2026

Learning from Demonstration in the Wild

arXiv 1811.03516 ↗First posted 8 Nov 2018cs.LG stat.MLcited by 25

On the map: Offline RL and imitation learning from data → Robot learning from demonstration

Abstract (arXiv)

Learning from demonstration (LfD) is useful in settings where hand-coding behaviour or a reward function is impractical. It has succeeded in a wide range of problems but typically relies on manually generated demonstrations or specially deployed sensors and has not generally been able to leverage the copious demonstrations available in the wild: those that capture behaviours that were occurring anyway using sensors that were already deployed for another purpose, e.g., traffic camera footage capturing demonstrations of natural behaviour of vehicles, cyclists, and pedestrians. We propose Video to Behaviour (ViBe), a new approach to learn models of behaviour from unlabelled raw video data of a traffic scene collected from a single, monocular, initially uncalibrated camera with ordinary resolution. Our approach calibrates the camera, detects relevant objects, tracks them through time, and uses the resulting trajectories to perform LfD, yielding models of naturalistic behaviour. We apply ViBe to raw videos of a traffic intersection and show that it can learn purely from videos, without additional expert knowledge.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 53 references that are in the map. First posted on arXiv 8 Nov 2018.

Builds on

25 of 53 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Mask R-CNNcited by 7,259
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2015Fast R-CNNcited by 2,930
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2016Generative Adversarial Imitation Learningcited by 1,576
  11. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  12. 2016Simple Online and Realtime Trackingcited by 862
13 more
  1. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  2. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  5. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  6. 2017One-Shot Imitation Learningcited by 232
  7. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  8. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  9. 2018Playing hard exploration games by watching YouTubecited by 98
  10. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  11. 2017Robust Imitation of Diverse Behaviorscited by 85
  12. 2016Learning by tracking: Siamese CNN for robust target associationcited by 79
  13. 2018Multi-Agent Imitation Learning for Driving Simulationcited by 49

Built on by

Cited by year

Citing papers per year: 2019: 2; 2020: 0; 2021: 4; 2022: 9; 2023: 7; 2024: 2; 2025: 1.

25 papers.

Sort:

By area:

  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  2. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  4. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  5. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  6. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
  7. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  8. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  9. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  10. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  11. 2022Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Modelscited by 27
  12. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
13 more
  1. 2019Independent Generative Adversarial Self-Imitation Learning in Cooperative Multiagent Systemscited by 7
  2. 2022No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RLcited by 6
  3. 2022Leveraging Sequentiality in Reinforcement Learning from a Single Demonstrationcited by 6
  4. 2022DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matchingcited by 5
  5. 2025DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learningcited by 5
  6. 2022Exploring the trade off between human driving imitation and safety for traffic simulationcited by 3
  7. 2023ConBaT: Control Barrier Transformer for Safe Policy Learningcited by 2
  8. 2023INTAGS: Interactive Agent-Guided Simulationcited by 1
  9. 2024LitSim: A Conflict-aware Policy for Long-term Interactive Traffic Simulationcited by 1
  10. 2021A Utility Maximization Model of Pedestrian and Driver Interactionscited by 0
  11. 2022Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interactioncited by 0
  12. 2023RObotic MAnipulation Network (ROMAN) -- Hybrid Hierarchical Learning for Solving Complex Sequential Taskscited by 0
  13. 2024Single-Reset Divide & Conquer Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Urban Driving with Conditional Imitation Learning
  2. 2026Imitation Learning for Autonomous Driving in CARLA
  3. 2022Model-Based Imitation Learning for Urban Driving
  4. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Video
  5. 2026Aligning Microscopic Vehicle and Macroscopic Traffic Statistics: Reconstructing Driving Behavior from Partial Data
  6. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  7. 2021Keyframe-Focused Visual Imitation Learning
  8. 2026Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation