mlsanityData to 14 Oct 2026

Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation

arXiv 2502.20420 ↗First posted 27 Feb 2025cs.CL cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

In this work, we provide the system description of our submission as part of the English to Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text only and Multimodal translation tasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 56 of its 85 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

56 of 85 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2015Fast R-CNNcited by 2,933
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  13. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  14. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  15. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  16. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  17. 2024LoRA Learns Less and Forgets Lesscited by 328
  18. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  19. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  20. 2020Experience Grounds Languagecited by 242
  21. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  22. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  23. 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
  24. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  25. 2017Imagination improves Multimodal Translationcited by 38
  26. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  27. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 32
  28. 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
  29. 2022On Vision Features in Multimodal Machine Translationcited by 31
  30. 2018The MeMAD Submission to the WMT18 Multimodal Translation Taskcited by 22
  31. 2016Multimodal Attention for Neural Machine Translationcited by 19
  32. 2019Generalizing Back-Translation in Neural Machine Translationcited by 18
  33. 2020Simultaneous Machine Translation with Visual Contextcited by 17
  34. 2019Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translationcited by 16
  35. 2022Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translationcited by 13
  36. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  37. 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
  38. 2017An empirical study on the effectiveness of images in Multimodal Neural Machine Translationcited by 10

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 8
  2. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptationcited by 3
  3. 2025Seeing Straight: Document Orientation Detection for Efficient OCRcited by 1
  4. 2026IndicFairFace: Balanced Indian Face Dataset for Auditing and Mitigating Geographical Bias in Vision-Language Modelscited by 0
  5. 2026Designing Production-Scale OCR for India: Multilingual and Domain-Specific Systemscited by 0
  6. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languagescited by 0