# Learning to Learn from Web Data through Deep Semantic Embeddings

arXiv: https://arxiv.org/abs/1808.06368 · HTML: https://mlsanity.com/paper/1808.06368/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2018.

## Built on by (most cited first, 5 of 5)

- [CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes](https://mlsanity.com/paper/2310.09761/) (2023)
- [Exploiting the relationship between visual and textual features in social networks for image classification with zero-shot deep learning](https://mlsanity.com/paper/2107.03751/) (2021)
- [Re-purposing Perceptual Hashing based Client Side Scanning for Physical Surveillance](https://mlsanity.com/paper/2212.04107/) (2022)
- [Weakly Supervised Deep Hyperspherical Quantization for Image Retrieval](https://mlsanity.com/paper/2404.04998/) (2024)
- [Beyond Absolute Scores: Relative Edit-induced Difference for Generalizable Image Aesthetic Assessment](https://mlsanity.com/paper/2606.05778/) (2026)

## Abstract (arXiv)

In this paper we propose to learn a multimodal image and text embedding from Web and Social Media data, aiming to leverage the semantic knowledge learnt in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the pipeline can learn from images with associated text without supervision and perform a thourough analysis of five different text embeddings in three different benchmarks. We show that the embeddings learnt with Web and Social Media data have competitive performances over supervised methods in the text based image retrieval task, and we clearly outperform state of the art in the MIRFlickr dataset when training in the target data. Further we demonstrate how semantic multimodal image retrieval can be performed using the learnt embeddings, going beyond classical instance-level retrieval problems. Finally, we present a new dataset, InstaCities1M, composed by Instagram images and their associated texts that can be used for fair comparison of image-text embeddings.