mlsanityData to 7 Oct 2026

Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training

arXiv 2303.05313 ↗First posted 9 Mar 2023cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Fine-grained supervision based on object annotations has been widely used for vision and language pre-training (VLP). However, in real-world application scenarios, aligned multi-modal data is usually in the image-caption format, which only provides coarse-grained supervision. It is not only cost-expensive but also compute-expensive to collect object annotations and build object annotation pre-extractor for different scenarios. In this paper, we propose a fine-grained VLP scheme without object annotations from the linguistic perspective. First, we propose a homonym sentence rewriting (HSR) algorithm to provide token-level supervision. The algorithm replaces a verb/noun/adjective/quantifier word of the caption with its homonyms from WordNet. Correspondingly, we propose refined vision-language modeling (RVLM) framework to exploit the token-level supervision. Three refined tasks, i.e., refined image-text contrastive (RITC), refined image-text matching (RITM), and replace language modeling (RLM) are proposed to learn the fine-grained alignment. Extensive experiments on several downstream tasks demonstrate the superior performance of the proposed method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

40 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
25 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  15. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  16. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  17. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  21. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  22. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  23. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  24. 2019Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 37
  25. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  3. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  4. 2024Anchor-based Robust Finetuning of Vision-Language Models
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  6. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  7. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
  8. 2023PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting