# Goal-driven text descriptions for images arXiv: https://arxiv.org/abs/2108.12575 ยท HTML: https://mlsanity.com/paper/2108.12575/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 122 of its 292 references that are in the map. - First posted on arXiv 28 Aug 2021. ## Builds on (most cited first, 30 of 122) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021) - [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016) - [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014) - [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015) - [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014) - [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016) - [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019) - [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013) - [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018) - [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016) - [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014) - [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018) - [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019) - [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020) ## Built on by (most cited first, 1 of 1) - [LLM4VG: Large Language Models Evaluation for Video Grounding](https://mlsanity.com/paper/2312.14206/) (2023) ## Abstract (arXiv) A big part of achieving Artificial General Intelligence(AGI) is to build a machine that can see and listen like humans. Much work has focused on designing models for image classification, video classification, object detection, pose estimation, speech recognition, etc., and has achieved significant progress in recent years thanks to deep learning. However, understanding the world is not enough. An AI agent also needs to know how to talk, especially how to communicate with a human. While perception (vision, for example) is more common across animal species, the use of complicated language is unique to humans and is one of the most important aspects of intelligence. In this thesis, we focus on generating textual output given visual input. In Chapter 3, we focus on generating the referring expression, a text description for an object in the image so that a receiver can infer which object is being described. We use a comprehension machine to directly guide the generated referring expressions to be more discriminative. In Chapter 4, we introduce a method that encourages discriminability in image caption generation. We show that more discriminative captioning models generate more descriptive captions. In Chapter 5, we study how training objectives and sampling methods affect the models' ability to generate diverse captions. We find that a popular captioning training strategy will be detrimental to the diversity of generated captions. In Chapter 6, we propose a model that can control the length of generated captions. By changing the desired length, one can influence the style and descriptiveness of the captions. Finally, in Chapter 7, we rank/generate informative image tags according to their information utility. The proposed method better matches what humans think are the most important tags for the images.