mlsanityData to 7 Oct 2026

OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understanding

arXiv 2510.00652 ↗First posted 1 Oct 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce OTTER, a unified open-set multi-label tagging framework that harmonizes the stability of a curated, predefined category set with the adaptability of user-driven open tags. OTTER is built upon a large-scale, hierarchically organized multi-modal dataset, collected from diverse online repositories and annotated through a hybrid pipeline combining automated vision-language labeling with human refinement. By leveraging a multi-head attention architecture, OTTER jointly aligns visual and textual representations with both fixed and open-set label embeddings, enabling dynamic and semantically consistent tagging. OTTER consistently outperforms competitive baselines on two benchmark datasets: it achieves an overall F1 score of 0.81 on Otter and 0.75 on Favorite, surpassing the next-best results by margins of 0.10 and 0.02, respectively. OTTER attains near-perfect performance on open-set labels, with F1 of 0.99 on Otter and 0.97 on Favorite, while maintaining competitive accuracy on predefined labels. These results demonstrate OTTER's effectiveness in bridging closed-set consistency with open-vocabulary flexibility for multi-modal tagging applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

24 of 40 references.

References · 24

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  6. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  7. 2018Bilinear Attention Networkscited by 277
  8. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  9. 2015Label-Embedding for Image Classificationcited by 164
  10. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  11. 2019Learning Semantic-Specific Graph Representation for Multi-Label Image Recognitioncited by 116
  12. 2017Multi-label Image Recognition by Recurrently Discovering Attentional Regionscited by 83
12 more
  1. 2020Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognitioncited by 80
  2. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  3. 2019Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classificationcited by 78
  4. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  5. 2020Learning to Discover Multi-Class Attentional Regions for Multi-Label Image Recognitioncited by 50
  6. 2015Exploit Bounding Box Annotations for Multi-label Object Recognitioncited by 39
  7. 2016Fast Zero-Shot Image Taggingcited by 21
  8. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Modelscited by 10
  9. 2024CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representationcited by 9
  10. 2022Open Vocabulary Multi-Label Classification with Dual-Modal Decoder on Aligned Visual-Textual Featurescited by 6
  11. 2024Data Processing Techniques for Modern Multimodal Modelscited by 3
  12. 2024Multimodal Multilabel Classification by CLIPcited by 2

Built on by

Not cited yet.