# METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidance

arXiv: https://arxiv.org/abs/2409.12667 · HTML: https://mlsanity.com/paper/2409.12667/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2308.00398/) (2023)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Hidden Biases of End-to-End Driving Models](https://mlsanity.com/paper/2306.07957/) (2023)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Embodied Understanding of Driving Scenarios](https://mlsanity.com/paper/2403.04593/) (2024)
- [VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes](https://mlsanity.com/paper/2405.05885/) (2024)
- [Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations](https://mlsanity.com/paper/2405.06039/) (2024)
- [Coaching a Teachable Student](https://mlsanity.com/paper/2306.10014/) (2023)
- [HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perception](https://mlsanity.com/paper/2201.06644/) (2022)
- [Transformer-Based Sensor Fusion for Autonomous Driving: A Survey](https://mlsanity.com/paper/2302.11481/) (2023)
- [HawkDrive: A Transformer-driven Visual Perception System for Autonomous Driving in Night Scene](https://mlsanity.com/paper/2404.04653/) (2024)
- [InteractionNet: Joint Planning and Prediction for Autonomous Driving with Transformers](https://mlsanity.com/paper/2309.03475/) (2023)
- [Perception Without Vision for Trajectory Prediction: Ego Vehicle Dynamics as Scene Representation for Efficient Active Learning in Autonomous Driving](https://mlsanity.com/paper/2405.09049/) (2024)
- [Towards Efficient Architecture and Algorithms for Sensor Fusion](https://mlsanity.com/paper/2209.06272/) (2022)

## Built on by (most cited first, 2 of 2)

- [Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations](https://mlsanity.com/paper/2603.11417/) (2026)
- [EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racing](https://mlsanity.com/paper/2604.11400/) (2026)

## Abstract (arXiv)

Multi-modal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system's understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.