# Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

arXiv: https://arxiv.org/abs/2601.22032 · HTML: https://mlsanity.com/paper/2601.22032/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 43 of its 47 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024)
- [Pseudo-Simulation for Autonomous Driving](https://mlsanity.com/paper/2506.04218/) (2025)
- [DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07656/) (2025)

## Built on by (most cited first, 30 of 30)

- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving](https://mlsanity.com/paper/2603.24581/) (2026)
- [Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation](https://mlsanity.com/paper/2606.15869/) (2026)
- [The DAWN of World-Action Interactive Models](https://mlsanity.com/paper/2605.11550/) (2026)
- [WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving](https://mlsanity.com/paper/2608.20974/) (2026)
- [Human Cognition in Machines: A Unified Perspective of World Models](https://mlsanity.com/paper/2604.16592/) (2026)
- [Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving](https://mlsanity.com/paper/2607.29031/) (2026)
- [DA-WAM: Decision-Aligned Future Latents for Driving World Models](https://mlsanity.com/paper/2608.19085/) (2026)
- [Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learning](https://mlsanity.com/paper/2606.05645/) (2026)
- [RISE: Adaptive Imagination for World Action Models](https://mlsanity.com/paper/2608.20430/) (2026)
- [Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations](https://mlsanity.com/paper/2603.11417/) (2026)
- [NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving](https://mlsanity.com/paper/2605.31116/) (2026)
- [CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving](https://mlsanity.com/paper/2606.06219/) (2026)
- [SUV: Future Scene Understanding as Video Generation for End-to-End Driving](https://mlsanity.com/paper/2608.03084/) (2026)
- [Zero-Label Driving Scenario Complexity Detection via Joint Embedding Predictive Architecture](https://mlsanity.com/paper/2606.28383/) (2026)
- [DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation](https://mlsanity.com/paper/2608.02428/) (2026)
- [Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving](https://mlsanity.com/paper/2608.30122/) (2026)
- [From Proxy Learning to Driving Decisions: A Transfer-Based Framework for Evaluating Future-Aware Autonomous Driving Planners](https://mlsanity.com/paper/2609.02688/) (2026)
- [READ: Learning Risk-Informed Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2609.12371/) (2026)
- [Planning-Aligned Pretraining of BEV Representations with Sparse Action-Conditioned Targets for End-to-End Autonomous Driving](https://mlsanity.com/paper/2609.22868/) (2026)
- [FeasibleFlow: One-Step Joint Transport of Configuration Feasibility and Trajectories for End-to-End Driving](https://mlsanity.com/paper/2609.23488/) (2026)
- [WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving](https://mlsanity.com/paper/2609.30436/) (2026)
- [ReDrive: Shaping Representations with World Modeling for End-to-End Driving](https://mlsanity.com/paper/2609.33854/) (2026)
- [AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving](https://mlsanity.com/paper/2609.34085/) (2026)
- [World4Scorer: Outcome-Grounded World Modeling for Autonomous Driving](https://mlsanity.com/paper/2609.36438/) (2026)
- [PhysWAM: Physically Consistent World Action Model for Autonomous Driving](https://mlsanity.com/paper/2609.37970/) (2026)
- [FPD-Drive: Future Policy Distillation for End-to-End Autonomous Driving](https://mlsanity.com/paper/2610.13232/) (2026)
- [MORE: Learning Motion-Oriented Registers for Driving World Models and Planning](https://mlsanity.com/paper/2610.14871/) (2026)
- [ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving](https://mlsanity.com/paper/2610.16908/) (2026)
- [VITAL: From Visual Motion to Precise Action](https://mlsanity.com/paper/2610.20604/) (2026)

## Abstract (arXiv)

End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.