# Fluency-Guided Cross-Lingual Image Captioning

arXiv: https://arxiv.org/abs/1708.04390 · HTML: https://mlsanity.com/paper/1708.04390/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2017.

## Built on by (most cited first, 30 of 33)

- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [CCMB: A Large-scale Chinese Cross-modal Benchmark](https://mlsanity.com/paper/2205.03860/) (2022)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [Multilingual Diversity Improves Vision-Language Representations](https://mlsanity.com/paper/2405.16915/) (2024)
- [LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation](https://mlsanity.com/paper/2411.04997/) (2024)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [Object-Centric Unsupervised Image Captioning](https://mlsanity.com/paper/2112.00969/) (2021)
- [Translation-Enhanced Multilingual Text-to-Image Generation](https://mlsanity.com/paper/2305.19216/) (2023)
- [MTVR: Multilingual Moment Retrieval in Videos](https://mlsanity.com/paper/2108.00061/) (2021)
- [CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation](https://mlsanity.com/paper/2407.01081/) (2024)
- [Bornon: Bengali Image Captioning with Transformer-based Deep learning approach](https://mlsanity.com/paper/2109.05218/) (2021)
- [ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation](https://mlsanity.com/paper/2303.06458/) (2023)
- [M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining](https://mlsanity.com/paper/2401.15896/) (2024)
- [Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution](https://mlsanity.com/paper/2505.10921/) (2025)
- [Enhancing Dynamic Image Advertising with Vision-Language Pre-training](https://mlsanity.com/paper/2306.14112/) (2023)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval](https://mlsanity.com/paper/2509.23370/) (2025)
- [Image captioning in different languages](https://mlsanity.com/paper/2407.09495/) (2024)
- [A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning](https://mlsanity.com/paper/2503.04592/) (2025)
- [Pre-Trained CNN Architecture for Transformer-Based Image Caption Generation Model](https://mlsanity.com/paper/2509.17365/) (2025)
- [DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset](https://mlsanity.com/paper/2601.10305/) (2026)

## Abstract (arXiv)

Image captioning has so far been explored mostly in English, as most available datasets are in this language. However, the application of image captioning should not be restricted by language. Only few studies have been conducted for image captioning in a cross-lingual setting. Different from these works that manually build a dataset for a target language, we aim to learn a cross-lingual captioning model fully from machine-translated sentences. To conquer the lack of fluency in the translated sentences, we propose in this paper a fluency-guided learning framework. The framework comprises a module to automatically estimate the fluency of the sentences and another module to utilize the estimated fluency scores to effectively train an image captioning model for the target language. As experiments on two bilingual (English-Chinese) datasets show, our approach improves both fluency and relevance of the generated captions in Chinese, but without using any manually written sentences from the target language.