# MacFormer: Map-Agent Coupled Transformer for Real-time and Robust Trajectory Prediction

arXiv: https://arxiv.org/abs/2308.10280 · HTML: https://mlsanity.com/paper/2308.10280/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)
- [Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding](https://mlsanity.com/paper/2003.03212/) (2020)
- [Learning to Predict Vehicle Trajectories with Model-based Planning](https://mlsanity.com/paper/2103.04027/) (2021)
- [GANet: Goal Area Network for Motion Forecasting](https://mlsanity.com/paper/2209.09723/) (2022)
- [Map-Adaptive Goal-Based Trajectory Prediction](https://mlsanity.com/paper/2009.04450/) (2020)
- [You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction](https://mlsanity.com/paper/2110.05304/) (2021)
- [MotionCNN: A Strong Baseline for Motion Prediction in Autonomous Driving](https://mlsanity.com/paper/2206.02163/) (2022)
- [Trajectory Prediction with Graph-based Dual-scale Context Fusion](https://mlsanity.com/paper/2111.01592/) (2021)

## Built on by (most cited first, 30 of 32)

- [HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention](https://mlsanity.com/paper/2404.06351/) (2024)
- [SEPT: Towards Efficient Scene Representation Learning for Motion Prediction](https://mlsanity.com/paper/2309.15289/) (2023)
- [SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving](https://mlsanity.com/paper/2402.02519/) (2024)
- [Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future](https://mlsanity.com/paper/2312.03408/) (2023)
- [KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation](https://mlsanity.com/paper/2407.12940/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Efficient Motion Prediction: A Lightweight & Accurate Trajectory Prediction Model With Fast Training and Inference Speed](https://mlsanity.com/paper/2409.16154/) (2024)
- [DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling](https://mlsanity.com/paper/2503.15029/) (2025)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction](https://mlsanity.com/paper/2506.21121/) (2025)
- [FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding](https://mlsanity.com/paper/2406.14422/) (2024)
- [Multi-Agent Trajectory Prediction with Difficulty-Guided Feature Enhancement Network](https://mlsanity.com/paper/2407.18551/) (2024)
- [Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics](https://mlsanity.com/paper/2507.12083/) (2025)
- [Future-Aware Interaction Network For Motion Forecasting](https://mlsanity.com/paper/2503.06565/) (2025)
- [Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting](https://mlsanity.com/paper/2402.03981/) (2024)
- [ProIn: Learning to Predict Trajectory Based on Progressive Interactions for Autonomous Driving](https://mlsanity.com/paper/2403.16374/) (2024)
- [Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models](https://mlsanity.com/paper/2407.19564/) (2024)
- [Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction](https://mlsanity.com/paper/2409.17385/) (2024)
- [CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention](https://mlsanity.com/paper/2409.17790/) (2024)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [A Structure-Aware Lane Graph Transformer Model for Vehicle Trajectory Prediction](https://mlsanity.com/paper/2405.20121/) (2024)
- [HAMF: A Hybrid Attention-Mamba Framework for Joint Scene Context Understanding and Future Motion Representation Learning](https://mlsanity.com/paper/2505.15703/) (2025)
- [Autoregressive Meta-Actions for Unified Controllable Trajectory Generation](https://mlsanity.com/paper/2505.23612/) (2025)
- [Post-interactive Multimodal Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2503.09366/) (2025)
- [LMFormer: Lane based Motion Prediction Transformer](https://mlsanity.com/paper/2504.10275/) (2025)
- [EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving](https://mlsanity.com/paper/2502.19260/) (2025)
- [GC-GAT: Multimodal Vehicular Trajectory Prediction using Graph Goal Conditioning and Cross-context Attention](https://mlsanity.com/paper/2504.11150/) (2025)
- [Trajectory Prediction in Dynamic Object Tracking: A Critical Study](https://mlsanity.com/paper/2506.19341/) (2025)
- [LTMSformer: A Local Trend-Aware Attention and Motion State Encoding Transformer for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2507.04634/) (2025)
- [ILNet: Trajectory Prediction with Inverse Learning Attention for Enhancing Intention Capture](https://mlsanity.com/paper/2507.06531/) (2025)

## Abstract (arXiv)

Predicting the future behavior of agents is a fundamental task in autonomous vehicle domains. Accurate prediction relies on comprehending the surrounding map, which significantly regularizes agent behaviors. However, existing methods have limitations in exploiting the map and exhibit a strong dependence on historical trajectories, which yield unsatisfactory prediction performance and robustness. Additionally, their heavy network architectures impede real-time applications. To tackle these problems, we propose Map-Agent Coupled Transformer (MacFormer) for real-time and robust trajectory prediction. Our framework explicitly incorporates map constraints into the network via two carefully designed modules named coupled map and reference extractor. A novel multi-task optimization strategy (MTOS) is presented to enhance learning of topology and rule constraints. We also devise bilateral query scheme in context fusion for a more efficient and lightweight network. We evaluated our approach on Argoverse 1, Argoverse 2, and nuScenes real-world benchmarks, where it all achieved state-of-the-art performance with the lowest inference latency and smallest model size. Experiments also demonstrate that our framework is resilient to imperfect tracklet inputs. Furthermore, we show that by combining with our proposed strategies, classical models outperform their baselines, further validating the versatility of our framework.