# STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset

arXiv: https://arxiv.org/abs/1705.00823 · HTML: https://mlsanity.com/paper/1705.00823/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2017.

## Built on by (most cited first, 30 of 47)

- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval](https://mlsanity.com/paper/2103.08784/) (2021)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training](https://mlsanity.com/paper/2206.00621/) (2022)
- [NLLB-CLIP -- train performant multilingual image retrieval model on a budget](https://mlsanity.com/paper/2309.01859/) (2023)
- [No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages](https://mlsanity.com/paper/2411.03769/) (2024)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval](https://mlsanity.com/paper/2409.19961/) (2024)
- [Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations](https://mlsanity.com/paper/2306.08658/) (2023)
- [CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2312.08984/) (2023)
- [MaXM: Towards Multilingual Visual Question Answering](https://mlsanity.com/paper/2209.05401/) (2022)
- [X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks](https://mlsanity.com/paper/2211.12402/) (2022)
- [mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus](https://mlsanity.com/paper/2406.08707/) (2024)
- [Semantic and Expressive Variation in Image Captions Across Languages](https://mlsanity.com/paper/2310.14356/) (2023)
- [CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation](https://mlsanity.com/paper/2407.01081/) (2024)
- [Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring](https://mlsanity.com/paper/2107.00372/) (2021)
- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [Improved Bengali Image Captioning via deep convolutional neural network based encoder-decoder model](https://mlsanity.com/paper/2102.07192/) (2021)
- [JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models](https://mlsanity.com/paper/2311.04192/) (2023)
- [Bornon: Bengali Image Captioning with Transformer-based Deep learning approach](https://mlsanity.com/paper/2109.05218/) (2021)
- [Cross-Lingual and Cross-Cultural Variation in Image Descriptions](https://mlsanity.com/paper/2409.16646/) (2024)
- [Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval](https://mlsanity.com/paper/2410.02027/) (2024)
- [MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs](https://mlsanity.com/paper/2505.19800/) (2025)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)

## Abstract (arXiv)

In recent years, automatic generation of image descriptions (captions), that is, image captioning, has attracted a great deal of attention. In this paper, we particularly consider generating Japanese captions for images. Since most available caption datasets have been constructed for English language, there are few datasets for Japanese. To tackle this problem, we construct a large-scale Japanese image caption dataset based on images from MS-COCO, which is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions for 164,062 images. In the experiment, we show that a neural network trained using STAIR Captions can generate more natural and better Japanese captions, compared to those generated using English-Japanese machine translation after generating English captions.