mlsanityData to 7 Oct 2026

Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder

arXiv 2311.08844 ↗First posted 15 Nov 2023cs.CV cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Although image captioning has a vast array of applications, it has not reached its full potential in languages other than English. Arabic, for instance, although the native language of more than 400 million people, remains largely underrepresented in this area. This is due to the lack of labeled data and powerful Arabic generative models. We alleviate this issue by presenting a novel vision-language model dedicated to Arabic, dubbed textitViolet. Our model is based on a vision encoder and a Gemini text decoder that maintains generation fluency while allowing fusion between the vision and language components. To train our model, we introduce a new method for automatically acquiring data from available English datasets. We also manually prepare a new dataset for evaluation. textitViolet performs sizeably better than our baselines on all of our evaluation datasets. For example, it reaches a CIDEr score of 61.2 on our manually annotated dataset and achieves an improvement of 13 points on Flickr8k.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 42 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

24 of 42 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
12 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2016Self-critical Sequence Training for Image Captioningcited by 544
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2019Attention on Attention for Image Captioningcited by 289
  7. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  8. 2016Image Captioning with Semantic Attentioncited by 260
  9. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  10. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  11. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  12. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 1.

7 papers.

  1. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarkscited by 32
  2. 2024Dallah: A Dialect-Aware Multimodal Large Language Model for Arabiccited by 16
  3. 2025JEEM: Vision-Language Understanding in Four Arabic Dialectscited by 7
  4. 2026Anthropogenic Regional Adaptation in Multimodal Vision-Language Modelcited by 2
  5. 2025Multimodal Arabic Captioning with Interpretable Visual Concept Integrationcited by 1
  6. 2024Image captioning in different languagescited by 0
  7. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  2. 2025Multilingual Training-Free Remote Sensing Image Captioning
  3. 2025Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR
  4. 2025AIN: The Arabic INclusive Large Multimodal Model
  5. 2025COCO-Urdu: A Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation
  6. 2026ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models