mlsanityData to 7 Oct 2026

Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers

arXiv 2501.08303 ↗First posted 14 Jan 2025cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. Project page and code at https://futurist-cvpr2025.github.io/ .

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 73 of its 87 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

73 of 87 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2024Depth Anything V2cited by 1,385
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  6. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  9. 2021Scaling Vision Transformerscited by 993
  10. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  11. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  13. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  14. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  15. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  16. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  17. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  18. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  19. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  20. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  21. 2016Generating Videos with Scene Dynamicscited by 376
  22. 2022MAGVIT: Masked Generative Video Transformercited by 341
  23. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  24. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  25. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  26. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  27. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  28. 2017Stochastic Variational Video Predictioncited by 234
  29. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  30. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  31. 2019Scaling Autoregressive Video Modelscited by 189
  32. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  33. 2018Stochastic Adversarial Video Predictioncited by 173
  34. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  35. 2018PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Modelcited by 133
  36. 20234M: Massively Multimodal Masked Modelingcited by 118
  37. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  38. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025A Survey of World Models for Autonomous Drivingcited by 60
  2. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
  3. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  4. 2026Representations Before Pixels: Semantics-Guided Hierarchical Video Predictioncited by 1
  5. 2026Future Dynamic 3D Reconstruction: Toward 3D World Modeling with Disentangled Ego-Motioncited by 1
  6. 2026Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learningcited by 0
  7. 2026Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Modelscited by 0