# Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge

arXiv: https://arxiv.org/abs/2012.11696 · HTML: https://mlsanity.com/paper/2012.11696/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 17 of 17)

- [DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models](https://mlsanity.com/paper/2210.06998/) (2022)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning](https://mlsanity.com/paper/2207.07635/) (2022)
- [Concadia: Towards Image-Based Text Generation with a Purpose](https://mlsanity.com/paper/2104.08376/) (2021)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [Multi-Modal Image Captioning for the Visually Impaired](https://mlsanity.com/paper/2105.08106/) (2021)
- [Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering](https://mlsanity.com/paper/2307.15745/) (2023)
- [StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI](https://mlsanity.com/paper/2508.08524/) (2025)
- [A Baseline for Detecting Out-of-Distribution Examples in Image Captioning](https://mlsanity.com/paper/2207.05418/) (2022)
- [From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation](https://mlsanity.com/paper/2311.12391/) (2023)
- [Towards Adaptable and Interactive Image Captioning with Data Augmentation and Episodic Memory](https://mlsanity.com/paper/2306.03500/) (2023)
- [Describing image focused in cognitive and visual details for visually impaired people: An approach to generating inclusive paragraphs](https://mlsanity.com/paper/2202.05331/) (2022)
- [Feedback is Needed for Retakes: An Explainable Poor Image Notification Framework for the Visually Impaired](https://mlsanity.com/paper/2211.09427/) (2022)
- [Compressed Image Captioning using CNN-based Encoder-Decoder Framework](https://mlsanity.com/paper/2404.18062/) (2024)
- [Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model](https://mlsanity.com/paper/2505.23358/) (2025)

## Abstract (arXiv)

Image captioning has recently demonstrated impressive progress largely owing to the introduction of neural network algorithms trained on curated dataset like MS-COCO. Often work in this field is motivated by the promise of deployment of captioning systems in practical applications. However, the scarcity of data and contexts in many competition datasets renders the utility of systems trained on these datasets limited as an assistive technology in real-world settings, such as helping visually impaired people navigate and accomplish everyday tasks. This gap motivated the introduction of the novel VizWiz dataset, which consists of images taken by the visually impaired and captions that have useful, task-oriented information. In an attempt to help the machine learning computer vision field realize its promise of producing technologies that have positive social impact, the curators of the VizWiz dataset host several competitions, including one for image captioning. This work details the theory and engineering from our winning submission to the 2020 captioning competition. Our work provides a step towards improved assistive image captioning systems.