# Visual Madlibs: Fill in the blank Image Generation and Question Answering

arXiv: https://arxiv.org/abs/1506.00278 · HTML: https://mlsanity.com/paper/1506.00278/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2015.

## Built on by (most cited first, 14 of 14)

- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs](https://mlsanity.com/paper/2101.12059/) (2021)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [A survey on VQA_Datasets and Approaches](https://mlsanity.com/paper/2105.00421/) (2021)
- [What Makes a Maze Look Like a Maze?](https://mlsanity.com/paper/2409.08202/) (2024)
- [VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery](https://mlsanity.com/paper/2509.17191/) (2025)
- [Show Why the Answer is Correct! Towards Explainable AI using Compositional Temporal Attention](https://mlsanity.com/paper/2105.07141/) (2021)
- [BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models](https://mlsanity.com/paper/2301.12032/) (2023)
- [Understanding eGFR Trajectories and Kidney Function Decline via Large Multimodal Models](https://mlsanity.com/paper/2409.02530/) (2024)

## Abstract (arXiv)

In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.