mlsanityData to 7 Oct 2026

End-to-end Contextual Perception and Prediction with Interaction Transformer

arXiv 2008.05927 ↗First posted 13 Aug 2020cs.CV cs.ROcited by 54

Not on the map. Why?

Abstract (arXiv)

In this paper, we tackle the problem of detecting objects in 3D and forecasting their future motion in the context of self-driving. Towards this goal, we design a novel approach that explicitly takes into account the interactions between actors. To capture their spatial-temporal dependencies, we propose a recurrent neural network with a novel Transformer architecture, which we call the Interaction Transformer. Importantly, our model can be trained end-to-end, and runs in real-time. We validate our approach on two challenging real-world datasets: ATG4D and nuScenes. We show that our approach can outperform the state-of-the-art on both datasets. In particular, we significantly improve the social compliance between the estimated future trajectories, resulting in far fewer collisions between the predicted actors.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 16; 2023: 8; 2024: 7; 2025: 7; 2026: 3.

54 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  4. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  5. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  6. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  7. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  8. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  9. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  10. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  11. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  12. 2021LaneRCNN: Distributed Representations for Graph-Centric Motion Forecastingcited by 112
38 more
  1. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  2. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  3. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  4. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  5. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  6. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  7. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  8. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  9. 2022SoMoFormer: Multi-Person Pose Forecasting with Transformerscited by 23
  10. 2021Deep Structured Reactive Planningcited by 18
  11. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Webcited by 16
  12. 2024Progressive Pretext Task Learning for Human Trajectory Predictioncited by 16
  13. 2024DeTra: A Unified Model for Object Detection and Trajectory Forecastingcited by 15
  14. 2022ParkPredict+: Multimodal Intent and Motion Prediction for Vehicles in Parking Lots with CNN and Transformercited by 13
  15. 2021Dyadic Human Motion Predictioncited by 9
  16. 2023Learning to Drive Anywherecited by 9
  17. 2024Efficient Motion Prediction: A Lightweight & Accurate Trajectory Prediction Model With Fast Training and Inference Speedcited by 9
  18. 2022Trajectory Forecasting from Detection with Uncertainty-Aware Motion Encodingcited by 8
  19. 2022Under the Hood of Transformer Networks for Trajectory Forecastingcited by 8
  20. 2023Attention-aware Social Graph Transformer Networks for Stochastic Trajectory Predictioncited by 8
  21. 2021Towards Robust Human Trajectory Prediction in Raw Videoscited by 7
  22. 2024Joint Perception and Prediction for Autonomous Driving: A Surveycited by 7
  23. 2022InterTrack: Interaction Transformer for 3D Multi-Object Trackingcited by 6
  24. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  25. 2023A Game-Theoretic Framework for Joint Forecasting and Planningcited by 5
  26. 2025ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Modelcited by 4
  27. 2023FFINet: Future Feedback Interaction Network for Motion Forecastingcited by 3
  28. 2024MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Drivingcited by 3
  29. 2024ITPNet: Towards Instantaneous Trajectory Prediction for Autonomous Drivingcited by 2
  30. 2026Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Predictioncited by 2
  31. 2022ProspectNet: Weighted Conditional Attention for Future Interaction Modeling in Behavior Predictioncited by 1
  32. 2022Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Drivingcited by 1
  33. 2023Cognitive TransFuser: Semantics-guided Transformer-based Sensor Fusion for Improved Waypoint Predictioncited by 1
  34. 2025To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Softwarecited by 1
  35. 2026Class-Incremental Motion Forecastingcited by 1
  36. 2026BEVPredFormer: Spatio-temporal Attention for BEV Instance Prediction in Autonomous Drivingcited by 1
  37. 2022Unified Simulation, Perception, and Generation of Human Behaviorcited by 0
  38. 2023Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehiclecited by 0

50 most cited of 54 All 54 (JSON)