# SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction

arXiv: https://arxiv.org/abs/2403.11492 · HTML: https://mlsanity.com/paper/2403.11492/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)
- [GANet: Goal Area Network for Motion Forecasting](https://mlsanity.com/paper/2209.09723/) (2022)
- [ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals](https://mlsanity.com/paper/2303.12071/) (2023)
- [Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction](https://mlsanity.com/paper/2305.14715/) (2023)
- [Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors](https://mlsanity.com/paper/2305.04412/) (2023)
- [R-Pred: Two-Stage Motion Prediction Via Tube-Query Attention-Based Trajectory Refinement](https://mlsanity.com/paper/2211.08609/) (2022)
- [Path-Aware Graph Attention for HD Maps in Motion Prediction](https://mlsanity.com/paper/2202.13772/) (2022)
- [Bootstrap Motion Forecasting With Self-Consistent Constraints](https://mlsanity.com/paper/2204.05859/) (2022)

## Built on by (most cited first, 30 of 34)

- [CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2502.19908/) (2025)
- [DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning](https://mlsanity.com/paper/2604.01765/) (2026)
- [Motion Forecasting in Continuous Driving](https://mlsanity.com/paper/2410.06007/) (2024)
- [Motion Forecasting for Autonomous Vehicles: A Survey](https://mlsanity.com/paper/2502.08664/) (2025)
- [SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction](https://mlsanity.com/paper/2410.08669/) (2024)
- [ModeSeq: Taming Sparse Multimodal Motion Prediction with Sequential Mode Modeling](https://mlsanity.com/paper/2411.11911/) (2024)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control](https://mlsanity.com/paper/2406.16038/) (2024)
- [Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model](https://mlsanity.com/paper/2507.22615/) (2025)
- [UniF2ace: A Unified Fine-grained Face Understanding and Generation Model](https://mlsanity.com/paper/2503.08120/) (2025)
- [ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2508.07089/) (2025)
- [StreamMOTP: Streaming and Unified Framework for Joint 3D Multi-Object Tracking and Trajectory Prediction](https://mlsanity.com/paper/2406.19844/) (2024)
- [Multi-scale Temporal Fusion Transformer for Incomplete Vehicle Trajectory Prediction](https://mlsanity.com/paper/2409.00904/) (2024)
- [GAMDTP: Dynamic Trajectory Prediction with Graph Attention Mamba Network](https://mlsanity.com/paper/2504.04862/) (2025)
- [HAMF: A Hybrid Attention-Mamba Framework for Joint Scene Context Understanding and Future Motion Representation Learning](https://mlsanity.com/paper/2505.15703/) (2025)
- [Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction](https://mlsanity.com/paper/2603.10597/) (2026)
- [LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving](https://mlsanity.com/paper/2604.08719/) (2026)
- [LMFormer: Lane based Motion Prediction Transformer](https://mlsanity.com/paper/2504.10275/) (2025)
- [Adaptive Output Steps: FlexiSteps Network for Dynamic Trajectory Prediction](https://mlsanity.com/paper/2508.17797/) (2025)
- [VISTA: A Vision and Intent-Aware Social Attention Framework for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2511.10203/) (2025)
- [HiMAP: History-aware Map-occupancy Prediction with Fallback](https://mlsanity.com/paper/2602.17231/) (2026)
- [Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling](https://mlsanity.com/paper/2603.01864/) (2026)
- [Ego-Human Motion Prediction with 3D-Aware LLM](https://mlsanity.com/paper/2607.07001/) (2026)
- [OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework](https://mlsanity.com/paper/2411.07711/) (2024)
- [ILNet: Trajectory Prediction with Inverse Learning Attention for Enhancing Intention Capture](https://mlsanity.com/paper/2507.06531/) (2025)
- [Ensemble of Pre-Trained Models for Long-Tailed Trajectory Prediction](https://mlsanity.com/paper/2509.13914/) (2025)
- [TaPD: Temporal-adaptive Progressive Distillation for Observation-Adaptive Trajectory Forecasting in Autonomous Driving](https://mlsanity.com/paper/2603.06231/) (2026)
- [MetaDAT: Generalizable Trajectory Prediction via Meta Pre-training and Data-Adaptive Test-Time Updating](https://mlsanity.com/paper/2603.09419/) (2026)
- [SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting](https://mlsanity.com/paper/2603.28091/) (2026)
- [Synthetic-to-Real Translation for Class-Agnostic Motion Prediction](https://mlsanity.com/paper/2607.06319/) (2026)

## Abstract (arXiv)

Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. Context information, such as road maps and surrounding agents' states, provides crucial geometric and semantic information for motion behavior prediction. To this end, recent works explore two-stage prediction frameworks where coarse trajectories are first proposed, and then used to select critical context information for trajectory refinement. However, they either incur a large amount of computation or bring limited improvement, if not both. In this paper, we introduce a novel scenario-adaptive refinement strategy, named SmartRefine, to refine prediction with minimal additional computation. Specifically, SmartRefine can comprehensively adapt refinement configurations based on each scenario's properties, and smartly chooses the number of refinement iterations by introducing a quality score to measure the prediction quality and remaining refinement potential of each scenario. SmartRefine is designed as a generic and flexible approach that can be seamlessly integrated into most state-of-the-art motion prediction models. Experiments on Argoverse (1 & 2) show that our method consistently improves the prediction accuracy of multiple state-of-the-art prediction models. Specifically, by adding SmartRefine to QCNet, we outperform all published ensemble-free works on the Argoverse 2 leaderboard (single agent track) at submission. Comprehensive studies are also conducted to ablate design choices and explore the mechanism behind multi-iteration refinement. Codes are available at https://github.com/opendilab/SmartRefine/