# DINO-Foresight: Looking into the Future with DINO

arXiv: https://arxiv.org/abs/2412.11673 · HTML: https://mlsanity.com/paper/2412.11673/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 98 of its 116 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 30 of 98)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)

## Built on by (most cited first, 30 of 45)

- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [Embodied AI Agents: Modeling the World](https://mlsanity.com/paper/2506.22355/) (2025)
- [Back to the Features: DINO as a Foundation for Video World Models](https://mlsanity.com/paper/2507.19468/) (2025)
- [Learning Latent Action World Models In The Wild](https://mlsanity.com/paper/2601.05230/) (2026)
- [The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2502.10498/) (2025)
- [NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction](https://mlsanity.com/paper/2512.01550/) (2025)
- [VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model](https://mlsanity.com/paper/2603.12655/) (2026)
- [VideoWorld 2: Learning Transferable Knowledge from Real-world Videos](https://mlsanity.com/paper/2602.10102/) (2026)
- [Closing the Train-Test Gap in World Models for Gradient-Based Planning](https://mlsanity.com/paper/2512.09929/) (2025)
- [Language-Conditioned World Modeling for Visual Navigation](https://mlsanity.com/paper/2603.26741/) (2026)
- [A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens](https://mlsanity.com/paper/2604.04913/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoning](https://mlsanity.com/paper/2508.09547/) (2025)
- [MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation](https://mlsanity.com/paper/2509.21797/) (2025)
- [Foundation Visual Encoders Are Secretly Few-Shot Anomaly Detectors](https://mlsanity.com/paper/2510.01934/) (2025)
- [SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation](https://mlsanity.com/paper/2512.03350/) (2025)
- [VFMF: World Modeling by Forecasting Vision Foundation Model Features](https://mlsanity.com/paper/2512.11225/) (2025)
- [REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion](https://mlsanity.com/paper/2512.16636/) (2025)
- [Frozen Forecasting: A Unified Evaluation](https://mlsanity.com/paper/2507.13942/) (2025)
- [DIP: Unsupervised Dense In-Context Post-training of Visual Representations](https://mlsanity.com/paper/2506.18463/) (2025)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving](https://mlsanity.com/paper/2603.03765/) (2026)
- [Envisioning the Future, One Step at a Time](https://mlsanity.com/paper/2604.09527/) (2026)
- [PhysiFormer: Learning to Simulate Mechanics in World Space](https://mlsanity.com/paper/2606.27364/) (2026)
- [MUFASA: A Multi-Layer Framework for Slot Attention](https://mlsanity.com/paper/2602.07544/) (2026)
- [Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction](https://mlsanity.com/paper/2604.11707/) (2026)
- [Learning Long-term Motion Embeddings for Efficient Kinematics Generation](https://mlsanity.com/paper/2604.11737/) (2026)
- [Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2606.10656/) (2026)
- [Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motion](https://mlsanity.com/paper/2606.18250/) (2026)

## Abstract (arXiv)

Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code at https://dino-foresight.github.io/ .