# Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation

arXiv: https://arxiv.org/abs/1706.06275 · HTML: https://mlsanity.com/paper/1706.06275/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2017.

## Built on by (most cited first, 5 of 5)

- [Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events](https://mlsanity.com/paper/2101.00822/) (2021)
- [CHAE: Fine-Grained Controllable Story Generation with Characters, Actions and Emotions](https://mlsanity.com/paper/2210.05221/) (2022)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning](https://mlsanity.com/paper/2503.04592/) (2025)
- [Using Deep Learning to Generate Semantically Correct Hindi Captions](https://mlsanity.com/paper/2602.13352/) (2026)

## Abstract (arXiv)

Recent work in computer vision has yielded impressive results in automatically describing images with natural language. Most of these systems generate captions in a sin- gle language, requiring multiple language-specific models to build a multilingual captioning system. We propose a very simple technique to build a single unified model across languages, using artificial tokens to control the language, making the captioning system more compact. We evaluate our approach on generating English and Japanese captions, and show that a typical neural captioning architecture is capable of learning a single model that can switch between two different languages.