# Automatic Concept Discovery from Parallel Text and Visual Corpora

arXiv: https://arxiv.org/abs/1509.07225 · HTML: https://mlsanity.com/paper/1509.07225/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2015.

## Built on by (most cited first, 10 of 10)

- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [Pre-trained Vision-Language Models Learn Discoverable Visual Concepts](https://mlsanity.com/paper/2404.12652/) (2024)
- [RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2411.01595/) (2024)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Bridging Gap between Image Pixels and Semantics via Supervision: A Survey](https://mlsanity.com/paper/2107.13757/) (2021)
- [Caption Generation on Scenes with Seen and Unseen Object Categories](https://mlsanity.com/paper/2108.06165/) (2021)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Neural Twins Talk & Alternative Calculations](https://mlsanity.com/paper/2108.02807/) (2021)

## Abstract (arXiv)

Humans connect language and vision to perceive the world. How to build a similar connection for computers? One possible way is via visual concepts, which are text terms that relate to visually discriminative entities. We propose an automatic visual concept discovery algorithm using parallel text and visual corpora; it filters text terms based on the visual discriminative power of the associated images, and groups them into concepts using visual and semantic similarities. We illustrate the applications of the discovered concepts using bidirectional image and sentence retrieval task and image tagging task, and show that the discovered concepts not only outperform several large sets of manually selected concepts significantly, but also achieves the state-of-the-art performance in the retrieval task.