# Open-Set Image Tagging with Multi-Grained Text Supervision

arXiv: https://arxiv.org/abs/2310.15200 · HTML: https://mlsanity.com/paper/2310.15200/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 63 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph](https://mlsanity.com/paper/2006.16934/) (2020)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)

## Built on by (most cited first, 30 of 67)

- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data](https://mlsanity.com/paper/2410.18558/) (2024)
- [Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models](https://mlsanity.com/paper/2606.03748/) (2026)
- [Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces](https://mlsanity.com/paper/2503.19199/) (2025)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance](https://mlsanity.com/paper/2402.08680/) (2024)
- [YOLOE: Real-Time Seeing Anything](https://mlsanity.com/paper/2503.07465/) (2025)
- [UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection](https://mlsanity.com/paper/2412.03342/) (2024)
- [X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning](https://mlsanity.com/paper/2508.07607/) (2025)
- [RWKV-CLIP: A Robust Vision-Language Representation Learner](https://mlsanity.com/paper/2406.06973/) (2024)
- [Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2502.02548/) (2025)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2407.09781/) (2024)
- [Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps](https://mlsanity.com/paper/2406.18115/) (2024)
- [CSAD: Unsupervised Component Segmentation for Logical Anomaly Detection](https://mlsanity.com/paper/2408.15628/) (2024)
- [Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models](https://mlsanity.com/paper/2409.15451/) (2024)
- [VideoOrion: Tokenizing Object Dynamics in Videos](https://mlsanity.com/paper/2411.16156/) (2024)
- [Visual Intention Grounding for Egocentric Assistants](https://mlsanity.com/paper/2504.13621/) (2025)
- [iMOVE: Instance-Motion-Aware Video Understanding](https://mlsanity.com/paper/2502.11594/) (2025)
- [InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention](https://mlsanity.com/paper/2509.16691/) (2025)
- [Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models](https://mlsanity.com/paper/2404.08030/) (2024)
- [Dense360: Dense Understanding from Omnidirectional Panoramas](https://mlsanity.com/paper/2506.14471/) (2025)
- [ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Tasks](https://mlsanity.com/paper/2508.08240/) (2025)
- [On Large Multimodal Models as Open-World Image Classifiers](https://mlsanity.com/paper/2503.21851/) (2025)
- [Unified Reinforcement and Imitation Learning for Vision-Language Models](https://mlsanity.com/paper/2510.19307/) (2025)
- [OCTO+: A Suite for Automatic Open-Vocabulary Object Placement in Mixed Reality](https://mlsanity.com/paper/2401.08973/) (2024)
- [MMR: Evaluating Reading Ability of Large Multimodal Models](https://mlsanity.com/paper/2408.14594/) (2024)
- [Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution](https://mlsanity.com/paper/2507.14367/) (2025)
- [Masking Teacher and Reinforcing Student for Distilling Vision-Language Models](https://mlsanity.com/paper/2512.22238/) (2025)
- [MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization](https://mlsanity.com/paper/2603.03192/) (2026)

## Abstract (arXiv)

In this paper, we introduce the Recognize Anything Plus Model (RAM++), an open-set image tagging model effectively leveraging multi-grained text supervision. Previous approaches (e.g., CLIP) primarily utilize global text supervision paired with images, leading to sub-optimal performance in recognizing multiple individual semantic tags. In contrast, RAM++ seamlessly integrates individual tag supervision with global text supervision, all within a unified alignment framework. This integration not only ensures efficient recognition of predefined tag categories, but also enhances generalization capabilities for diverse open-set categories. Furthermore, RAM++ employs large language models (LLMs) to convert semantically constrained tag supervision into more expansive tag description supervision, thereby enriching the scope of open-set visual description concepts. Comprehensive evaluations on various image recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art (SOTA) open-set image tagging models on most aspects. Specifically, for predefined commonly used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP enhancements over CLIP on OpenImages and ImageNet. For open-set categories beyond predefined, RAM++ records improvements of 5.0 mAP and 6.4 mAP over CLIP and RAM respectively on OpenImages. For diverse human-object interaction phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark. Code, datasets and pre-trained models are available at urlhttps://github.com/xinyu1205/recognize-anything.