mlsanityData to 7 Oct 2026

IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training

arXiv 2207.05333 ↗First posted 12 Jul 2022cs.CV cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pre-training (VLP) with large-scale image-text pairs has demonstrated superior performance in various fields. However, the image-text pairs co-occurrent on the Internet typically lack explicit alignment information, which is suboptimal for VLP. Existing methods proposed to adopt an off-the-shelf object detector to utilize additional image tag information. However, the object detector is time-consuming and can only identify the pre-defined object categories, limiting the model capacity. Inspired by the observation that the texts incorporate incomplete fine-grained image information, we introduce IDEA, which stands for increasing text diversity via online multi-label recognition for VLP. IDEA shows that multi-label learning with image tags extracted from the texts can be jointly optimized during VLP. Moreover, IDEA can identify valuable image tags online to provide more explicit textual supervision. Comprehensive experiments demonstrate that IDEA can significantly boost the performance on multiple downstream datasets with a small extra computational cost.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 55 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

38 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
23 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  9. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  10. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  13. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  14. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  15. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  16. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  17. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  18. 2019Learning a Deep ConvNet for Multi-label Classification with Partial Labelscited by 124
  19. 2021Multi-Label Learning from Single Positive Labelscited by 81
  20. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  21. 2021Simple and Robust Loss Design for Multi-Label Learning with Missing Labelscited by 30
  22. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  23. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 7; 2024: 5; 2025: 1.

14 papers.

Sort:

By area:

  1. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  2. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  3. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  4. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  5. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
  6. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 19
  7. 2023LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognitioncited by 15
  8. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  9. 2023Unsupervised Prototype Adapter for Vision-Language Modelscited by 5
  10. 2024Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tagscited by 2
  11. 2024TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learningcited by 2
  12. 2024MOSMOS: Multi-organ segmentation facilitated by medical report supervisioncited by 1
2 more
  1. 2024A Survey on Incomplete Multi-label Learning: Recent Advances and Future Trendscited by 0
  2. 2025The impact of abstract and object tags on image privacy classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text as Image: Learning Transferable Adapter for Multi-Label Classification
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  3. 2023COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment
  4. 2023Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training
  5. 2022Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting
  6. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  7. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognition
  8. 2022Vision-Language Pre-Training for Boosting Scene Text Detectors