# Learning Visual N-Grams from Web Data

arXiv: https://arxiv.org/abs/1612.09161 · HTML: https://mlsanity.com/paper/1612.09161/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2016.

## Built on by (most cited first, 30 of 95)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://mlsanity.com/paper/2112.09583/) (2021)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Image Captioners Are Scalable Vision Learners Too](https://mlsanity.com/paper/2306.07915/) (2023)
- [Debiased Learning from Naturally Imbalanced Pseudo-Labels](https://mlsanity.com/paper/2201.01490/) (2022)
- [K-LITE: Learning Transferable Visual Models with External Knowledge](https://mlsanity.com/paper/2204.09222/) (2022)
- [An Inverse Scaling Law for CLIP Training](https://mlsanity.com/paper/2305.07017/) (2023)
- [Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search](https://mlsanity.com/paper/2102.01645/) (2021)

## Abstract (arXiv)

Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.