# OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts

arXiv: https://arxiv.org/abs/1707.07102 · HTML: https://mlsanity.com/paper/1707.07102/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2017.

## Built on by (most cited first, 8 of 8)

- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs](https://mlsanity.com/paper/2101.12059/) (2021)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [End-to-End Video Question-Answer Generation with Generator-Pretester Network](https://mlsanity.com/paper/2101.01447/) (2021)
- [Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Review](https://mlsanity.com/paper/2212.03747/) (2022)
- [CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion](https://mlsanity.com/paper/2404.00412/) (2024)
- [Resilience through Scene Context in Visual Referring Expression Generation](https://mlsanity.com/paper/2404.12289/) (2024)
- [Caption Generation on Scenes with Seen and Unseen Object Categories](https://mlsanity.com/paper/2108.06165/) (2021)

## Abstract (arXiv)

Generating captions for images is a task that has recently received considerable attention. In this work we focus on caption generation for abstract scenes, or object layouts where the only information provided is a set of objects and their locations. We propose OBJ2TEXT, a sequence-to-sequence model that encodes a set of objects and their locations as an input sequence using an LSTM network, and decodes this representation using an LSTM language model. We show that our model, despite encoding object layouts as a sequence, can represent spatial relationships between objects, and generate descriptions that are globally coherent and semantically relevant. We test our approach in a task of object-layout captioning by using only object annotations as inputs. We additionally show that our model, combined with a state-of-the-art object detector, improves an image captioning model from 0.863 to 0.950 (CIDEr score) in the test benchmark of the standard MS-COCO Captioning task.