# Separating Self-Expression and Visual Content in Hashtag Supervision

arXiv: https://arxiv.org/abs/1711.09825 · HTML: https://mlsanity.com/paper/1711.09825/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 9 of 9)

- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information](https://mlsanity.com/paper/2211.09807/) (2022)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning](https://mlsanity.com/paper/2406.07543/) (2024)
- [Large-Scale Attribute-Object Compositions](https://mlsanity.com/paper/2105.11373/) (2021)
- [EUFCC-340K: A Faceted Hierarchical Dataset for Metadata Annotation in GLAM Collections](https://mlsanity.com/paper/2406.02380/) (2024)
- [Utilizing a digital swarm intelligence platform to improve consensus among radiologists and exploring its applications](https://mlsanity.com/paper/2107.07341/) (2021)
- [Token-free Models for Sarcasm Detection](https://mlsanity.com/paper/2505.01006/) (2025)

## Abstract (arXiv)

The variety, abundance, and structured nature of hashtags make them an interesting data source for training vision models. For instance, hashtags have the potential to significantly reduce the problem of manual supervision and annotation when learning vision models for a large number of concepts. However, a key challenge when learning from hashtags is that they are inherently subjective because they are provided by users as a form of self-expression. As a consequence, hashtags may have synonyms (different hashtags referring to the same visual content) and may be ambiguous (the same hashtag referring to different visual content). These challenges limit the effectiveness of approaches that simply treat hashtags as image-label pairs. This paper presents an approach that extends upon modeling simple image-label pairs by modeling the joint distribution of images, hashtags, and users. We demonstrate the efficacy of such approaches in image tagging and retrieval experiments, and show how the joint model can be used to perform user-conditional retrieval and tagging.