mlsanityData to 7 Oct 2026

Tag2Text: Guiding Vision-Language Model via Image Tagging

arXiv 2303.05657 ↗First posted 10 Mar 2023cs.CVcited by 84

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shelf detector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot performance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance. Code, demo and pre-trained models are available at https://github.com/xinyu1205/recognize-anything.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 73 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

56 of 73 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training12 Jul
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  5. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  15. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  16. 2016Self-critical Sequence Training for Image Captioningcited by 544
  17. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  20. 2018nocaps: novel object captioning at scalecited by 444
  21. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  22. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  23. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  24. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  25. 2015Visual Semantic Role Labelingcited by 188
  26. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  27. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  28. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  29. 2014From Captions to Visual Concepts and Backcited by 109
  30. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  31. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  32. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  33. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  34. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  35. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  36. 2022Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfercited by 33
  37. 2021Simple and Robust Loss Design for Multi-Label Learning with Missing Labelscited by 30
  38. 2018Intention Oriented Image Captions with Guiding Objectscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 32; 2025: 22; 2026: 10.

84 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  7. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  8. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  9. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  10. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  11. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  12. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
38 more
  1. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  2. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  3. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  4. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  5. 2024Hawk: Learning to Understand Open-World Video Anomaliescited by 47
  6. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  7. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 40
  8. 2024MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generationcited by 38
  9. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  10. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 26
  11. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  12. 2024Self-Guidance: Boosting Flow and Diffusion Generation on Their Owncited by 22
  13. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  14. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understandingcited by 15
  15. 2024Training-Free Open-Ended Object Detection and Segmentation via Attention as Promptscited by 15
  16. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 14
  17. 2024ControlCap: Controllable Region-level Captioningcited by 13
  18. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  19. 2023Object Recognition as Next Token Predictioncited by 12
  20. 2025FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noisecited by 11
  21. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10
  22. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  23. 2023PRIME: Prioritizing Interpretability in Failure Mode Extractioncited by 9
  24. 2025Towards Scalable and Consistent 3D Editingcited by 9
  25. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  26. 2025AMD-Hummingbird: Towards an Efficient Text-to-Video Modelcited by 8
  27. 2024TRINS: Towards Multimodal Language Models that Can Readcited by 7
  28. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understandingcited by 6
  29. 2024Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversioncited by 6
  30. 2025DualMap: Online Open-Vocabulary Semantic Mapping for Natural Language Navigation in Dynamic Changing Scenescited by 6
  31. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classificationcited by 5
  32. 2024Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Costcited by 5
  33. 2023RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignmentcited by 4
  34. 2024Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarioscited by 4
  35. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  36. 2025AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Pathcited by 4
  37. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
  38. 2023Text as Image: Learning Transferable Adapter for Multi-Label Classificationcited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Are Multimodal Large Language Models Good Annotators for Image Tagging?
  2. 2025Text-Guided Semantic Image Encoder
  3. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
  4. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
  5. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Models
  6. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
  7. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  8. 2024Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes