mlsanityData to 14 Oct 2026

Merlin:Empowering Multimodal LLMs with Foresight Minds

arXiv 2312.00589 ↗First posted 30 Nov 2023cs.CVcited by 46

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Humans possess the remarkable ability to foresee the future to a certain extent based on present observations, a skill we term as foresight minds. However, this capability remains largely under explored within existing Multimodal Large Language Models (MLLMs), hindering their capacity to learn the fundamental principles of how things operate and the intentions behind the observed subjects. To address this issue, we introduce the integration of future modeling into the existing learning frameworks of MLLMs. By utilizing the subject trajectory, a highly structured representation of a consecutive frame sequence, as a learning objective, we aim to bridge the gap between the past and the future. We propose two innovative methods to empower MLLMs with foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning (FIT), which are inspired by the modern learning paradigm of LLMs. Specifically, FPT jointly training various tasks centered on trajectories, enabling MLLMs to learn how to attend and predict entire trajectories from a given initial observation. Then, FIT requires MLLMs to first predict trajectories of related objects and then reason about potential future events based on them. Aided by FPT and FIT, we build a novel and unified MLLM named Merlin that supports multi-images input and analysis about potential actions of multiple objects for the future reasoning. Experimental results show Merlin powerful foresight minds with impressive performance on both future reasoning and visual comprehension tasks.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 80 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

64 of 80 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  8. 2023Generating Images with Multimodal Language Models26 May
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  10. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2022Token Merging: Your ViT But Fastercited by 1,021
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  15. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  16. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  17. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 568
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  19. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 532
  22. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  23. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 446
  24. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  25. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  26. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  27. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  28. 2018CrowdHuman: A Benchmark for Detecting Human in a Crowdcited by 284
  29. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  30. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  31. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  32. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  33. 2021DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motioncited by 217
  34. 2019SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelinescited by 177
  35. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  36. 2023SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenescited by 131
  37. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  38. 2023DetGPT: Detect What You Need via Reasoningcited by 112

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 21; 2026: 9.

46 papers.

Sort:

By area:

  1. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  3. 2023Video Understanding with Large Language Models: A Surveycited by 206
  4. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  5. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  6. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  7. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  8. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  9. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  10. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  11. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  12. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
34 more
  1. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  2. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  3. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  4. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  5. 20253D Aware Region Prompted Vision Language Modelcited by 33
  6. 2026STEP3-VL-10B Technical Reportcited by 32
  7. 2024Self-Supervised Visual Preference Alignmentcited by 31
  8. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  9. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  10. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  11. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?cited by 20
  12. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  13. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 18
  14. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  15. 2025Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Drivingcited by 16
  16. 2025Perception in Reflectioncited by 13
  17. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  18. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  19. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  20. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  21. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 8
  22. 2024Uncertainty-boosted Robust Video Activity Anticipationcited by 7
  23. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understandingcited by 7
  24. 2025Disentangling Instance and Scene Contexts for 3D Semantic Scene Completioncited by 7
  25. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  26. 2026GREx: Generalized Referring Expression Segmentation, Comprehension, and Generationcited by 6
  27. 2025RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Casecited by 4
  28. 2025ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Trackingcited by 3
  29. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 2
  30. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  31. 2026FeVOS: Foresight Expression Video Object Segmentationcited by 0
  32. 2026MentalThink: Shaping Thoughts in Mental SVG Worldcited by 0
  33. 2026Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Trackingcited by 0
  34. 2026VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
  2. 2025Thinking Ahead: Foresight Intelligence in MLLMs and World Model
  3. 2026ForeAct: Steering Your VLA with Efficient Visual Foresight Planning
  4. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions
  5. 2024From Efficient Multimodal Models to World Models: A Survey
  6. 2026Foresight: planning future perception in streaming VLMs without retraining
  7. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMs
  8. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning