mlsanityData to 7 Oct 2026

DINO-Foresight: Looking into the Future with DINO

arXiv 2412.11673 ↗First posted 16 Dec 2024cs.CVcited by 44

Not on the map. Why?

Cited by 20 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code at https://dino-foresight.github.io/ .

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 98 of its 116 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

98 of 116 references.

References · 98

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023Segment Anythingcited by 8,158
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2016Pyramid Scene Parsing Networkcited by 2,472
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2021Vision Transformers for Dense Predictioncited by 1,789
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2024Depth Anything V2cited by 1,385
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  13. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  14. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  15. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  18. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  19. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  20. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  21. 2023Vision Transformers Need Registerscited by 844
  22. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  23. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  24. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  25. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  26. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  27. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  28. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  29. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  30. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  31. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  32. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  33. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  34. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  35. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  36. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  37. 2016Generating Videos with Scene Dynamicscited by 376
  38. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 29.

44 papers.

Sort:

By area:

  1. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  2. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  3. 2025Embodied AI Agents: Modeling the Worldcited by 57
  4. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  5. 2026Learning Latent Action World Models In The Wildcited by 42
  6. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  7. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  8. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  9. 2026VGGT-World: Transforming VGGT into an Autoregressive Geometry World Modelcited by 11
  10. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planningcited by 10
  11. 2026Language-Conditioned World Modeling for Visual Navigationcited by 9
  12. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokenscited by 9
32 more
  1. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  2. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoningcited by 6
  3. 2025MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulationcited by 5
  4. 2025Foundation Visual Encoders Are Secretly Few-Shot Anomaly Detectorscited by 5
  5. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Featurescited by 5
  6. 2025REGLUE Your Latents with Global and Local Semantics for Entangled Diffusioncited by 5
  7. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  8. 2025DIP: Unsupervised Dense In-Context Post-training of Visual Representationscited by 3
  9. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  10. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  11. 2026LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Drivingcited by 2
  12. 2026Envisioning the Future, One Step at a Timecited by 2
  13. 2026MUFASA: A Multi-Layer Framework for Slot Attentioncited by 1
  14. 2026Representations Before Pixels: Semantics-Guided Hierarchical Video Predictioncited by 1
  15. 2026Learning Long-term Motion Embeddings for Efficient Kinematics Generationcited by 1
  16. 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Drivingcited by 1
  17. 2026Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motioncited by 1
  18. 2026PhysiFormer: Learning to Simulate Mechanics in World Spacecited by 1
  19. 2026Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Compositioncited by 0
  20. 2026Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learningcited by 0
  21. 2026Coevolving Representations in Joint Image-Feature Diffusioncited by 0
  22. 2026Divide and Conquer: Decoupled Representation Alignment for Multimodal World Modelscited by 0
  23. 2026LACO: Adaptive Latent Communication for Collaborative Drivingcited by 0
  24. 2026The Perception-Physics Paradox: Probing Scientific Alignment with TC-Benchcited by 0
  25. 2026DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Predictioncited by 0
  26. 2026DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigationcited by 0
  27. 2026Luce: Relightable Gaussians for 3D Asset Generationcited by 0
  28. 2026ViBR-WM: Visual Bayesian Regression for World Modelingcited by 0
  29. 2026Where Predictive Supervision Goes Shapes What VLA Policies Learncited by 0
  30. 2026Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradientscited by 0
  31. 2026Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulationcited by 0
  32. 2026Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Modelscited by 0