# Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos

arXiv: https://arxiv.org/abs/2602.22091 · HTML: https://mlsanity.com/paper/2602.22091/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors](https://mlsanity.com/paper/2412.12392/) (2024)
- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [Visual Point Cloud Forecasting enables Scalable Autonomous Driving](https://mlsanity.com/paper/2312.17655/) (2023)
- [DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input](https://mlsanity.com/paper/2409.12753/) (2024)
- [Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining](https://mlsanity.com/paper/2204.02393/) (2022)
- [Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling](https://mlsanity.com/paper/2301.01006/) (2023)
- [SelfD: Self-Learning Large-Scale Driving Policies From the Web](https://mlsanity.com/paper/2204.10320/) (2022)
- [VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving](https://mlsanity.com/paper/2411.14716/) (2024)

## Built on by (most cited first, 3 of 3)

- [IDOL: Inverse-Dynamics-Guided Future Prediction for End-to-End Autonomous Driving](https://mlsanity.com/paper/2605.31476/) (2026)
- [Planning in the Backbone: DiffAdapterVLA for Native Continuous Trajectory Generation with Driving VLMs](https://mlsanity.com/paper/2609.15322/) (2026)
- [ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving](https://mlsanity.com/paper/2610.16908/) (2026)

## Abstract (arXiv)

Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.