mlsanityData to 7 Oct 2026

Learning by Watching

arXiv 2106.05966 ↗First posted 10 Jun 2021cs.CV cs.AI cs.LGcited by 22

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

When in a new situation or geographical location, human drivers have an extraordinary ability to watch others and learn maneuvers that they themselves may have never performed. In contrast, existing techniques for learning to drive preclude such a possibility as they assume direct access to an instrumented ego-vehicle with fully known observations and expert driver actions. However, such measurements cannot be directly accessed for the non-ego vehicles when learning by watching others. Therefore, in an application where data is regarded as a highly valuable asset, current approaches completely discard the vast portion of the training data that can be potentially obtained through indirect observation of surrounding vehicles. Motivated by this key insight, we propose the Learning by Watching (LbW) framework which enables learning a driving policy without requiring full knowledge of neither the state nor expert actions. To increase its data, i.e., with new perspectives and maneuvers, LbW makes use of the demonstrations of other vehicles in a given scene by (1) transforming the ego-vehicle's observations to their points of view, and (2) inferring their expert actions. Our LbW agent learns more robust driving policies while enabling data-efficient learning, including quick adaptation of the policy to rare and novel scenarios. In particular, LbW drives robustly even with a fraction of available driving data required by existing methods, achieving an average success rate of 92% on the original CARLA benchmark with only 30 minutes of total driving data and 82% with only 10 minutes.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 54 of its 80 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

54 of 80 references.

References · 54

  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  5. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  6. 2016End to End Learning for Self-Driving Carscited by 1,000
  7. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  8. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  10. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  11. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  12. 2019Learning by Cheatingcited by 361
38 more
  1. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  2. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  3. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  4. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  5. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  6. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  7. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  8. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
  9. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  10. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 173
  11. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  12. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  13. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  14. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  15. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  16. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  17. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  18. 2017Third-Person Imitation Learningcited by 103
  19. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  20. 2018Driving Policy Transfer via Modularity and Abstractioncited by 91
  21. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  22. 2018Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networkscited by 84
  23. 2019Urban Driving with Conditional Imitation Learningcited by 84
  24. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  25. 2020A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye Viewcited by 71
  26. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  27. 2019What You See is What You Get: Exploiting Visibility for 3D Object Detectioncited by 69
  28. 2019Does computer vision matter for action?cited by 64
  29. 2020FISHING Net: Future Inference of Semantic Heatmaps In Gridscited by 55
  30. 2020MonoLayout: Amodal scene layout from a single imagecited by 49
  31. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  32. 2020Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Drivingcited by 41
  33. 2018Learning to Look around Objects for Top-View Representations of Outdoor Scenescited by 37
  34. 2019Monocular Plan View Networks for Autonomous Drivingcited by 37
  35. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  36. 2019A Geometric Approach to Obtain a Bird's Eye View from an Imagecited by 34
  37. 2020Label Efficient Visual Abstractions for Autonomous Drivingcited by 33
  38. 2018Generative Adversarial Frontal View to Bird View Synthesiscited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 5; 2024: 4; 2025: 2; 2026: 2.

22 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  3. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  4. 2022Learning from All Vehiclescited by 135
  5. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  6. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  7. 2023Coaching a Teachable Studentcited by 17
  8. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 17
  9. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 16
  10. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  11. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  12. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
10 more
  1. 2024On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Drivingcited by 10
  2. 2021Causal Imitative Model for Autonomous Drivingcited by 7
  3. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  4. 2022Learn-to-Race Challenge 2022: Benchmarking Safe Learning and Cross-domain Generalisation in Autonomous Racingcited by 4
  5. 2023AutoGraph: Predicting Lane Graphs from Traffic Observationscited by 4
  6. 2024Trajectory Planning for Autonomous Driving in Unstructured Scenarios Based on Graph Neural Network and Numerical Optimizationcited by 1
  7. 2021Designed to Cooperate: A Kant-Inspired Ethic of Machine-to-Machine Cooperationcited by 0
  8. 2025Learning to Drive by Imitating Surrounding Vehiclescited by 0
  9. 2026Mirror Learningcited by 0
  10. 2026SkyDrive: Learning to Drive in a New City from Aerial Traffic Monitoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
  2. 2021Towards Learning Generalizable Driving Policies from Restricted Latent Representations
  3. 2025Learning to Drive from a World Model
  4. 2026Aligning Microscopic Vehicle and Macroscopic Traffic Statistics: Reconstructing Driving Behavior from Partial Data
  5. 2021Learning to drive from a world on rails
  6. 2026DriveZero: End-to-End Driving Beyond Human Demonstrations
  7. 2026What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planning
  8. 2026TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations