# AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks

arXiv: https://arxiv.org/abs/2509.16438 · HTML: https://mlsanity.com/paper/2509.16438/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [Speech Recognition Challenge in the Wild: Arabic MGB-3](https://mlsanity.com/paper/1709.07276/) (2017)
- [The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition](https://mlsanity.com/paper/1609.05625/) (2016)
- [Arabic natural language processing: An overview](https://mlsanity.com/paper/1903.02784/) (2019)
- [ArabicaQA: A Comprehensive Dataset for Arabic Question Answering](https://mlsanity.com/paper/2403.17848/) (2024)
- [Rudder: A Cross Lingual Video and Text Retrieval Dataset](https://mlsanity.com/paper/2103.05457/) (2021)
- [Towards Arabic Multimodal Dataset for Sentiment Analysis](https://mlsanity.com/paper/2306.06322/) (2023)
- [MSVD-Indonesian: A Benchmark for Multimodal Video-Text Tasks in Indonesian](https://mlsanity.com/paper/2306.11341/) (2023)

## Abstract (arXiv)

Video-to-text and text-to-video retrieval are dominated by English benchmarks (e.g. DiDeMo, MSR-VTT) and recent multilingual corpora (e.g. RUDDER), yet Arabic remains underserved, lacking localized evaluation metrics. We introduce a three-stage framework, AutoArabic, utilizing state-of-the-art large language models (LLMs) to translate non-Arabic benchmarks into Modern Standard Arabic, reducing the manual revision required by nearly fourfold. The framework incorporates an error detection module that automatically flags potential translation errors with 97% accuracy. Applying the framework to DiDeMo, a video retrieval benchmark produces DiDeMo-AR, an Arabic variant with 40,144 fluent Arabic descriptions. An analysis of the translation errors is provided and organized into an insightful taxonomy to guide future Arabic localization efforts. We train a CLIP-style baseline with identical hyperparameters on the Arabic and English variants of the benchmark, finding a moderate performance gap (about 3 percentage points at Recall@1), indicating that Arabic localization preserves benchmark difficulty. We evaluate three post-editing budgets (zero/ flagged-only/ full) and find that performance improves monotonically with more post-editing, while the raw LLM output (zero-budget) remains usable. To ensure reproducibility to other languages, we made the code available at https://github.com/Tahaalshatiri/AutoArabic.