# Learning to Caption Images through a Lifetime by Asking Questions

arXiv: https://arxiv.org/abs/1812.00235 · HTML: https://mlsanity.com/paper/1812.00235/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2018.

## Built on by (most cited first, 11 of 11)

- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [Few-Shot Continual Active Learning by a Robot](https://mlsanity.com/paper/2210.04137/) (2022)
- [Single-Modal Entropy based Active Learning for Visual Question Answering](https://mlsanity.com/paper/2110.10906/) (2021)
- [Interactive Machine Learning for Image Captioning](https://mlsanity.com/paper/2202.13623/) (2022)
- [Learning Novel Objects Continually Through Curiosity](https://mlsanity.com/paper/2103.07758/) (2021)
- [Learning by Asking Questions for Knowledge-based Novel Object Recognition](https://mlsanity.com/paper/2210.05879/) (2022)
- [Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation](https://mlsanity.com/paper/2401.10005/) (2024)
- [Towards Adaptable and Interactive Image Captioning with Data Augmentation and Episodic Memory](https://mlsanity.com/paper/2306.03500/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Continual Learning of Visual Concepts for Robots through Limited Supervision](https://mlsanity.com/paper/2101.10509/) (2021)
- [Putting Humans in the Image Captioning Loop](https://mlsanity.com/paper/2306.03476/) (2023)

## Abstract (arXiv)

In order to bring artificial agents into our lives, we will need to go beyond supervised learning on closed datasets to having the ability to continuously expand knowledge. Inspired by a student learning in a classroom, we present an agent that can continuously learn by posing natural language questions to humans. Our agent is composed of three interacting modules, one that performs captioning, another that generates questions and a decision maker that learns when to ask questions by implicitly reasoning about the uncertainty of the agent and expertise of the teacher. As compared to current active learning methods which query images for full captions, our agent is able to ask pointed questions to improve the generated captions. The agent trains on the improved captions, expanding its knowledge. We show that our approach achieves better performance using less human supervision than the baselines on the challenging MSCOCO dataset.