mlsanityData to 7 Oct 2026

Fine-Grained Semantically Aligned Vision-Language Pre-Training

arXiv 2208.02515 ↗First posted 4 Aug 2022cs.CVcited by 68

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced cross-modal attention upon image and text features. However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available. In this paper, we introduce LOUPE, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions. To efficiently compute the game-theoretic interactions, we further propose an uncertainty-aware neural Shapley interaction learning module. Experiments show that LOUPE achieves state-of-the-art performance on a variety of vision-language tasks. Furthermore, without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding. More importantly, LOUPE opens a new promising direction of learning fine-grained semantics from large-scale raw image-text pairs. The repository of this work is at https://github.com/YYJMJC/LOUPE.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 72 references that are in the map. First posted on arXiv 4 Aug 2022.

Builds on

45 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  5. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2018YOLOv3: An Incremental Improvementcited by 2,652
28 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2016Modeling Context in Referring Expressionscited by 1,054
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  15. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  18. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  19. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  20. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  21. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  22. 2018Zero-Shot Object Detectioncited by 189
  23. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  24. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  25. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  26. 2019Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigationcited by 39
  27. 2022Adapting CLIP For Phrase Localization Without Further Trainingcited by 21
  28. 2021A Unified Game-Theoretic Interpretation of Adversarial Robustnesscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 22; 2024: 21; 2025: 14; 2026: 6.

68 papers.

Sort:

By area:

  1. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  2. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  3. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  4. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  5. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  6. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  7. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  8. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  9. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  10. 2023UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularitycited by 30
  11. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  12. 2022ISS: Image as Stepping Stone for Text-Guided 3D Shape Generationcited by 27
38 more
  1. 2023Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusioncited by 27
  2. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  3. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  4. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Modelscited by 21
  5. 2023G2L: Semantically Aligned and Uniform Video Grounding via Geodesic and Game Theorycited by 21
  6. 2022VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignmentcited by 20
  7. 2023Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Groundingcited by 19
  8. 2024Towards Unified Multimodal Editing with Enhanced Knowledge Collaborationcited by 18
  9. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15
  10. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  11. 2025Multi-Granular Multimodal Clue Fusion for Meme Understandingcited by 14
  12. 2024PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysiscited by 13
  13. 2024IPO: Interpretable Prompt Optimization for Vision-Language Modelscited by 13
  14. 2023Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaborationcited by 12
  15. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  16. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Modelscited by 12
  17. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  18. 2022Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correctioncited by 10
  19. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  20. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  21. 2023ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Datacited by 9
  22. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  23. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  24. 2026Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspectivecited by 8
  25. 2023Improving Vision Anomaly Detection with the Guidance of Language Modalitycited by 7
  26. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7
  27. 2024Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refiningcited by 7
  28. 2025WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localizationcited by 7
  29. 2023TG-VQA: Ternary Game of Video Question Answeringcited by 6
  30. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  31. 2024Semantic Codebook Learning for Dynamic Recommendation Modelscited by 6
  32. 2022TIER: Text-Image Entropy Regularization for CLIP-style modelscited by 5
  33. 2023I3: Intent-Introspective Retrieval Conditioned on Instructionscited by 5
  34. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  35. 2025VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semanticscited by 5
  36. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  37. 2024Hierarchical Banzhaf Interaction for General Video-Language Representation Learningcited by 4
  38. 2025VinaBench: Benchmark for Faithful and Consistent Visual Narrativescited by 4

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  2. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  3. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
  6. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
  7. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
  8. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis