# GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting

arXiv: https://arxiv.org/abs/2211.02545 · HTML: https://mlsanity.com/paper/2211.02545/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding](https://mlsanity.com/paper/2205.09753/) (2022)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans](https://mlsanity.com/paper/2001.00735/) (2020)
- [DSDNet: Deep Structured self-Driving Network](https://mlsanity.com/paper/2008.06041/) (2020)
- [End-to-end Contextual Perception and Prediction with Interaction Transformer](https://mlsanity.com/paper/2008.05927/) (2020)

## Built on by (most cited first, 30 of 68)

- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders](https://mlsanity.com/paper/2308.09882/) (2023)
- [ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals](https://mlsanity.com/paper/2303.12071/) (2023)
- [FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs](https://mlsanity.com/paper/2211.16197/) (2022)
- [Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding](https://mlsanity.com/paper/2310.12970/) (2023)
- [Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving](https://mlsanity.com/paper/2308.01471/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving](https://mlsanity.com/paper/2402.02519/) (2024)
- [SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction](https://mlsanity.com/paper/2405.15677/) (2024)
- [Learning Realistic Traffic Agents in Closed-loop](https://mlsanity.com/paper/2311.01394/) (2023)
- [Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?](https://mlsanity.com/paper/2306.09281/) (2023)
- [Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSM](https://mlsanity.com/paper/2503.10898/) (2025)
- [DeTra: A Unified Model for Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2406.04426/) (2024)
- [TrafficBots V1.5: Traffic Simulation via Conditional VAEs and Transformers with Relative Pose Encoding](https://mlsanity.com/paper/2406.10898/) (2024)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [Motion Forecasting in Continuous Driving](https://mlsanity.com/paper/2410.06007/) (2024)
- [Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation](https://mlsanity.com/paper/2311.01446/) (2023)
- [Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent](https://mlsanity.com/paper/2311.18307/) (2023)
- [Efficient Motion Prediction: A Lightweight & Accurate Trajectory Prediction Model With Fast Training and Inference Speed](https://mlsanity.com/paper/2409.16154/) (2024)
- [BiFF: Bi-level Future Fusion with Polyline-based Coordinate for Interactive Trajectory Prediction](https://mlsanity.com/paper/2306.14161/) (2023)
- [CRITERIA: a New Benchmarking Paradigm for Evaluating Trajectory Prediction Models for Autonomous Driving](https://mlsanity.com/paper/2310.07794/) (2023)
- [Conditional Unscented Autoencoders for Trajectory Prediction](https://mlsanity.com/paper/2310.19944/) (2023)
- [LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds](https://mlsanity.com/paper/2311.01444/) (2023)
- [JointMotion: Joint Self-Supervision for Joint Motion Prediction](https://mlsanity.com/paper/2403.05489/) (2024)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [RedMotion: Motion Prediction via Redundancy Reduction](https://mlsanity.com/paper/2306.10840/) (2023)
- [FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding](https://mlsanity.com/paper/2406.14422/) (2024)
- [Multi-Agent Trajectory Prediction with Difficulty-Guided Feature Enhancement Network](https://mlsanity.com/paper/2407.18551/) (2024)

## Abstract (arXiv)

The task of motion forecasting is critical for self-driving vehicles (SDVs) to be able to plan a safe maneuver. Towards this goal, modern approaches reason about the map, the agents' past trajectories and their interactions in order to produce accurate forecasts. The predominant approach has been to encode the map and other agents in the reference frame of each target agent. However, this approach is computationally expensive for multi-agent prediction as inference needs to be run for each agent. To tackle the scaling challenge, the solution thus far has been to encode all agents and the map in a shared coordinate frame (e.g., the SDV frame). However, this is sample inefficient and vulnerable to domain shift (e.g., when the SDV visits uncommon states). In contrast, in this paper, we propose an efficient shared encoding for all agents and the map without sacrificing accuracy or generalization. Towards this goal, we leverage pair-wise relative positional encodings to represent geometric relationships between the agents and the map elements in a heterogeneous spatial graph. This parameterization allows us to be invariant to scene viewpoint, and save online computation by re-using map embeddings computed offline. Our decoder is also viewpoint agnostic, predicting agent goals on the lane graph to enable diverse and context-aware multimodal prediction. We demonstrate the effectiveness of our approach on the urban Argoverse 2 benchmark as well as a novel highway dataset.