# What value do explicit high level concepts have in vision to language problems?

arXiv: https://arxiv.org/abs/1506.01144 · HTML: https://mlsanity.com/paper/1506.01144/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2015.

## Built on by (most cited first, 30 of 31)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition](https://mlsanity.com/paper/2112.06482/) (2021)
- [Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment](https://mlsanity.com/paper/2208.13628/) (2022)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Exploring Discrete Diffusion Models for Image Captioning](https://mlsanity.com/paper/2211.11694/) (2022)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [A Survey of Natural Language Generation](https://mlsanity.com/paper/2112.11739/) (2021)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [Boosting Video-Text Retrieval with Explicit High-Level Semantics](https://mlsanity.com/paper/2208.04215/) (2022)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [FACT: Learning Governing Abstractions Behind Integer Sequences](https://mlsanity.com/paper/2209.09543/) (2022)
- [Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning](https://mlsanity.com/paper/2405.02766/) (2024)
- [DescribeEarth: Describe Anything for Remote Sensing Images](https://mlsanity.com/paper/2509.25654/) (2025)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention](https://mlsanity.com/paper/2210.16431/) (2022)
- [Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags](https://mlsanity.com/paper/2406.10839/) (2024)
- [Coreset Selection via LLM-based Concept Bottlenecks](https://mlsanity.com/paper/2502.16733/) (2025)
- [A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception](https://mlsanity.com/paper/2208.05757/) (2022)
- [How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation](https://mlsanity.com/paper/2211.10992/) (2022)
- [Pixels to Prose: Understanding the art of Image Captioning](https://mlsanity.com/paper/2408.15714/) (2024)
- [RefineCap: Concept-Aware Refinement for Image Captioning](https://mlsanity.com/paper/2109.03529/) (2021)
- [An Integrated Approach for Video Captioning and Applications](https://mlsanity.com/paper/2201.09153/) (2022)
- [An Image captioning algorithm based on the Hybrid Deep Learning Technique (CNN+GRU)](https://mlsanity.com/paper/2301.02440/) (2023)
- [Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning](https://mlsanity.com/paper/2302.04676/) (2023)

## Abstract (arXiv)

Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. We propose here a method of incorporating high-level concepts into the very successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art performance in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. In doing so we provide an analysis of the value of high level semantic information in V2L problems.