# Boosting Image Captioning with Attributes

arXiv: https://arxiv.org/abs/1611.01646 · HTML: https://mlsanity.com/paper/1611.01646/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2016.

## Built on by (most cited first, 30 of 76)

- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning](https://mlsanity.com/paper/2311.07574/) (2023)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022)
- [CPTR: Full Transformer Network for Image Captioning](https://mlsanity.com/paper/2101.10804/) (2021)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022)
- [M^4I: Multi-modal Models Membership Inference](https://mlsanity.com/paper/2209.06997/) (2022)
- [Comprehending and Ordering Semantics for Image Captioning](https://mlsanity.com/paper/2206.06930/) (2022)
- [Conversational Fashion Image Retrieval via Multiturn Natural Language Feedback](https://mlsanity.com/paper/2106.04128/) (2021)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning](https://mlsanity.com/paper/2108.02366/) (2021)
- [Towards Local Visual Modeling for Image Captioning](https://mlsanity.com/paper/2302.06098/) (2023)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation](https://mlsanity.com/paper/2301.09209/) (2023)
- [Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network](https://mlsanity.com/paper/2101.11562/) (2021)
- [Detector-Free Weakly Supervised Grounding by Separation](https://mlsanity.com/paper/2104.09829/) (2021)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [Exploring Discrete Diffusion Models for Image Captioning](https://mlsanity.com/paper/2211.11694/) (2022)
- [Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval](https://mlsanity.com/paper/2210.08908/) (2022)
- [SGNet: A Super-class Guided Network for Image Classification and Object Detection](https://mlsanity.com/paper/2104.12898/) (2021)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics](https://mlsanity.com/paper/2108.08217/) (2021)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [Boosting Video-Text Retrieval with Explicit High-Level Semantics](https://mlsanity.com/paper/2208.04215/) (2022)

## Abstract (arXiv)

Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.