# Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge

arXiv: https://arxiv.org/abs/2306.15700 · HTML: https://mlsanity.com/paper/2306.15700/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds](https://mlsanity.com/paper/2112.09205/) (2021)
- [HOPE: Hierarchical Spatial-temporal Network for Occupancy Flow Prediction](https://mlsanity.com/paper/2206.10118/) (2022)

## Built on by (most cited first, 19 of 19)

- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mamba](https://mlsanity.com/paper/2408.03601/) (2024)
- [Centaur: Robust End-to-End Autonomous Driving with Test-Time Training](https://mlsanity.com/paper/2503.11650/) (2025)
- [Solving Motion Planning Tasks with a Scalable Generative Model](https://mlsanity.com/paper/2407.02797/) (2024)
- [PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning](https://mlsanity.com/paper/2406.01587/) (2024)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Rethinking Imitation-based Planner for Autonomous Driving](https://mlsanity.com/paper/2309.10443/) (2023)
- [Generalizing Motion Planners with Mixture of Experts for Autonomous Driving](https://mlsanity.com/paper/2410.15774/) (2024)
- [Hybrid-Prediction Integrated Planning for Autonomous Driving](https://mlsanity.com/paper/2402.02426/) (2024)
- [End-To-End Planning of Autonomous Driving in Industry and Academia: 2022-2023](https://mlsanity.com/paper/2401.08658/) (2023)
- [LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends](https://mlsanity.com/paper/2410.15281/) (2024)
- [Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving](https://mlsanity.com/paper/2409.15730/) (2024)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [Spatially Visual Perception for End-to-End Robotic Learning](https://mlsanity.com/paper/2411.17458/) (2024)
- [DeepUrban: Interaction-Aware Trajectory Prediction and Planning for Automated Driving by Aerial Imagery](https://mlsanity.com/paper/2601.10554/) (2026)
- [Utilizing Navigation Paths to Generate Target Points for Enhanced End-to-End Autonomous Driving Planning](https://mlsanity.com/paper/2406.08349/) (2024)
- [CAFE-AD: Cross-Scenario Adaptive Feature Enhancement for Trajectory Planning in Autonomous Driving](https://mlsanity.com/paper/2504.06584/) (2025)
- [PlanTRansformer: Unified Prediction and Planning with Goal-conditioned Transformer](https://mlsanity.com/paper/2602.03376/) (2026)
- [Sequence of Expert: Boosting Imitation Planners for Autonomous Driving through Temporal Alternation](https://mlsanity.com/paper/2512.13094/) (2025)

## Abstract (arXiv)

This paper presents our 2nd place solution for the NuPlan Challenge 2023. Autonomous driving in real-world scenarios is highly complex and uncertain. Achieving safe planning in the complex multimodal scenarios is a highly challenging task. Our approach, Imitation with Spatial-Temporal Heatmap, adopts the learning form of behavior cloning, innovatively predicts the future multimodal states with a heatmap representation, and uses trajectory refinement techniques to ensure final safety. The experiment shows that our method effectively balances the vehicle's progress and safety, generating safe and comfortable trajectories. In the NuPlan competition, we achieved the second highest overall score, while obtained the best scores in the ego progress and comfort metrics.