# Exploiting the relationship between visual and textual features in social networks for image classification with zero-shot deep learning

arXiv: https://arxiv.org/abs/2107.03751 · HTML: https://mlsanity.com/paper/2107.03751/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 14 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 5 of 5)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [Transferable Contrastive Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/1908.05832/) (2019)
- [Learning to Learn from Web Data through Deep Semantic Embeddings](https://mlsanity.com/paper/1808.06368/) (2018)

## Built on by (most cited first, 2 of 2)

- [Detecting Calls to Action in Multimodal Content: Analysis of the 2021 German Federal Election Campaign on Instagram](https://mlsanity.com/paper/2409.02690/) (2024)
- [Label Distribution Learning-Enhanced Dual-KNN for Text Classification](https://mlsanity.com/paper/2503.04869/) (2025)

## Abstract (arXiv)

One of the main issues related to unsupervised machine learning is the cost of processing and extracting useful information from large datasets. In this work, we propose a classifier ensemble based on the transferable learning capabilities of the CLIP neural network architecture in multimodal environments (image and text) from social media. For this purpose, we used the InstaNY100K dataset and proposed a validation approach based on sampling techniques. Our experiments, based on image classification tasks according to the labels of the Places dataset, are performed by first considering only the visual part, and then adding the associated texts as support. The results obtained demonstrated that trained neural networks such as CLIP can be successfully applied to image classification with little fine-tuning, and considering the associated texts to the images can help to improve the accuracy depending on the goal. The results demonstrated what seems to be a promising research direction.