Multimodal Multilabel Classification by CLIP
Not on the map. Why?
Abstract (arXiv)
Multimodal multilabel classification (MMC) is a challenging task that aims to design a learning algorithm to handle two data sources, the image and text, and learn a comprehensive semantic feature presentation across the modalities. In this task, we review the extensive number of state-of-the-art approaches in MMC and leverage a novel technique that utilises the Contrastive Language-Image Pre-training (CLIP) as the feature extractor and fine-tune the model by exploring different classification heads, fusion methods and loss functions. Finally, our best result achieved more than 90% F_1 score in the public Kaggle competition leaderboard. This paper provides detailed descriptions of novel training methods and quantitative analysis through the experimental results.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 23 Jun 2024.
Builds on
28 of 39 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2014Going Deeper with Convolutionscited by 4,231
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
16 more
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
- 2018A Survey on Deep Transfer Learningcited by 417
- 2018Stacked Cross Attention for Image-Text Matchingcited by 414
- 2020Asymmetric Loss For Multi-Label Classificationcited by 379
- 2021Pay Attention to MLPscited by 367
- 2017Snapshot Ensembles: Train 1, get M for freecited by 349
- 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
- 2020Local-Global Video-Text Interactions for Temporal Groundingcited by 188
- 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
- 2016Modelling Radiological Language with Bidirectional Long Short-Term Memory Networkscited by 9
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.