mlsanityData to 7 Oct 2026

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

arXiv 2101.06547 ↗First posted 16 Jan 2021cs.RO cs.AI cs.CVcited by 96

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

In this paper, we present LookOut, a novel autonomy system that perceives the environment, predicts a diverse set of futures of how the scene might unroll and estimates the trajectory of the SDV by optimizing a set of contingency plans over these future realizations. In particular, we learn a diverse joint distribution over multi-agent future trajectories in a traffic scene that covers a wide range of future modes with high sample efficiency while leveraging the expressive power of generative models. Unlike previous work in diverse motion forecasting, our diversity objective explicitly rewards sampling future scenarios that require distinct reactions from the self-driving vehicle for improved safety. Our contingency planner then finds comfortable and non-conservative trajectories that ensure safe reactions to a wide range of future scenarios. Through extensive evaluations, we show that our model demonstrates significantly more diverse and sample-efficient motion forecasting in a large-scale self-driving dataset as well as safer and less-conservative motion plans in long-term closed-loop simulations when compared to current state-of-the-art models.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 37 of its 58 references that are in the map. First posted on arXiv 16 Jan 2021.

Builds on

37 of 58 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Group Normalizationcited by 1,514
  3. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  4. 2016End to End Learning for Self-Driving Carscited by 1,000
  5. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  6. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  7. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  8. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  9. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  10. 2018Learning to Drive in a Daycited by 266
  11. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  12. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 251
25 more
  1. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  2. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  3. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  4. 2019Multiple Futures Predictioncited by 178
  5. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  6. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  7. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  8. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  9. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  10. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  11. 2018Driving Policy Transfer via Modularity and Abstractioncited by 91
  12. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  13. 2020HDNET: Exploiting HD Maps for 3D Object Detectioncited by 78
  14. 2016Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networkscited by 72
  15. 2020Diverse and Admissible Trajectory Forecasting through Multimodal Context Understandingcited by 68
  16. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  17. 20163D Fully Convolutional Network for Vehicle Detection in Point Cloudcited by 57
  18. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  19. 2019DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Samplingcited by 40
  20. 2019Discrete Residual Flow for Probabilistic Pedestrian Behavior Predictioncited by 34
  21. 2020Diversity inducing Information Bottleneck in Model Ensemblescited by 27
  22. 2018Search-based optimal motion planning for automated drivingcited by 24
  23. 2018An Auto-tuning Framework for Autonomous Vehiclescited by 5
  24. 2018Improving Rotated Text Detection with Rotation Region Proposal Networkscited by 3
  25. 2020Decision-Time Postponing Motion Planning for Combinatorial Uncertain Maneuveringcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 19; 2023: 28; 2024: 20; 2025: 18; 2026: 9.

96 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  4. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  5. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  6. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  7. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  8. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  9. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  10. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  11. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  12. 2022Learning from All Vehiclescited by 135
38 more
  1. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  2. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  3. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  4. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  5. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  6. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  7. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  8. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  9. 2022ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planningcited by 63
  10. 2025A Survey of World Models for Autonomous Drivingcited by 60
  11. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  12. 2023DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Drivingcited by 46
  13. 2022FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphscited by 41
  14. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  15. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 40
  16. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehiclescited by 36
  17. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  18. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  19. 2023Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encodingcited by 32
  20. 2023Implicit Occupancy Flow Fields for Perception and Prediction in Self-Drivingcited by 30
  21. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  22. 2022Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Expertscited by 28
  23. 2024SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Predictioncited by 25
  24. 2021Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Modelscited by 22
  25. 2024Generalizing Motion Planners with Mixture of Experts for Autonomous Drivingcited by 22
  26. 2022LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Predictioncited by 19
  27. 2024DeTra: A Unified Model for Object Detection and Trajectory Forecastingcited by 15
  28. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  29. 2022Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecastingcited by 14
  30. 2023Interactive Joint Planning for Autonomous Vehiclescited by 13
  31. 2022RAP: Risk-Aware Prediction for Robust Planningcited by 12
  32. 2024CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Modelcited by 12
  33. 2025FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedbackcited by 12
  34. 2024QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Drivingcited by 11
  35. 2023EPG-MGCN: Ego-Planning Guided Multi-Graph Convolutional Network for Heterogeneous Agent Trajectory Predictioncited by 10
  36. 2024On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Drivingcited by 10
  37. 2024Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Drivingcited by 9
  38. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution
  2. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  3. 2025CoPlanner: An Interactive Motion Planner with Contingency-Aware Diffusion for Autonomous Driving
  4. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectories
  5. 2026Bridging Predictive Uncertainty and Safe Action: Sample-Conditioned Differentiable Planning for Autonomous Driving
  6. 2026SparseWorld: Enhancing End-to-End Autonomous Driving via World Models with Sparse Scene Representation
  7. 2024Diffusion-Based Environment-Aware Trajectory Prediction
  8. 2025FutureX: Enhance End-to-End Autonomous Driving via Latent Chain-of-Thought World Model