mlsanityData to 7 Oct 2026

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

arXiv 2604.08337 ↗First posted 9 Apr 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current vision-language pre-training (VLP) paradigms excel at global scene understanding but struggle with instance-level reasoning due to global-only supervision. We introduce InstAP, an Instance-Aware Pre-training framework that jointly optimizes global vision-text alignment and fine-grained, instance-level contrastive alignment by grounding textual mentions to specific spatial-temporal regions. To support this, we present InstVL, a large-scale dataset (2 million images, 50,000 videos) with dual-granularity annotations: holistic scene captions and dense, grounded instance descriptions. On the InstVL benchmark, InstAP substantially outperforms existing VLP models on instance-level retrieval, and also surpasses a strong VLP baseline trained on the exact same data corpus, isolating the benefit of our instance-aware objective. Moreover, instance-centric pre-training improves global understanding: InstAP achieves competitive zero-shot performance on multiple video benchmarks, including MSR-VTT and DiDeMo. Qualitative visualizations further show that InstAP localizes textual mentions to the correct instances, while global-only models exhibit more diffuse, scene-level attention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 72 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

60 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Training4 Aug
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov

Other references · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
38 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2017Localizing Moments in Video with Natural Languagecited by 660
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  13. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  14. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  15. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  16. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  17. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  18. 2023Data Filtering Networkscited by 247
  19. 2022Egocentric Video-Language Pretrainingcited by 231
  20. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  21. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  22. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  24. 2016Movie Descriptioncited by 180
  25. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  26. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  27. 2022Multiview Transformers for Video Recognitioncited by 148
  28. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  29. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  30. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  31. 2018Grounded Video Descriptioncited by 78
  32. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  33. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  34. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  35. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  36. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  37. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  38. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Locality Alignment Improves Vision-Language Models
  2. 2024Toward Interactive Regional Understanding in Vision-Large Language Models
  3. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  4. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  5. 2026From Pixels to Words -- Towards Native One-Vision Models at Scale
  6. 2025Dynamic Context-Aware Scene Reasoning Using Vision-Language Alignment in Zero-Shot Real-World Scenarios
  7. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
  8. 2025Perception Encoder: The best visual embeddings are not at the output of the network