# Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation

arXiv: https://arxiv.org/abs/2109.04653 · HTML: https://mlsanity.com/paper/2109.04653/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Analyzing the Behavior of Visual Question Answering Models](https://mlsanity.com/paper/1606.07356/) (2016)
- [Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering](https://mlsanity.com/paper/1708.01471/) (2017)
- [Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering](https://mlsanity.com/paper/1505.05612/) (2015)
- [Compact Trilinear Interaction for Visual Question Answering](https://mlsanity.com/paper/1909.11874/) (2019)
- [Unsupervised Multilingual Word Embeddings](https://mlsanity.com/paper/1808.08933/) (2018)
- [Code-Switching Language Modeling using Syntax-Aware Multi-Task Learning](https://mlsanity.com/paper/1805.12070/) (2018)
- [Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training](https://mlsanity.com/paper/1810.11895/) (2018)
- [Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation](https://mlsanity.com/paper/2104.05965/) (2021)
- [Complexity Metric for Code-Mixed Social Media Text](https://mlsanity.com/paper/1707.01183/) (2017)
- [SMPOST: Parts of Speech Tagger for Code-Mixed Indic Social Media Text](https://mlsanity.com/paper/1702.00167/) (2017)

## Built on by (most cited first, 13 of 13)

- [MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering](https://mlsanity.com/paper/2405.11985/) (2024)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [EVJVQA Challenge: Multilingual Visual Question Answering](https://mlsanity.com/paper/2302.11752/) (2023)
- [Improving the Cross-Lingual Generalisation in Visual Question Answering](https://mlsanity.com/paper/2209.02982/) (2022)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Non-Autoregressive Sign Language Production via Knowledge Distillation](https://mlsanity.com/paper/2208.06183/) (2022)
- [C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval](https://mlsanity.com/paper/2210.03625/) (2022)
- [The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering](https://mlsanity.com/paper/2501.07109/) (2025)
- [Marathi-English Code-mixed Text Generation](https://mlsanity.com/paper/2309.16202/) (2023)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches](https://mlsanity.com/paper/2309.12224/) (2023)
- [Curriculum Script Distillation for Multilingual Visual Question Answering](https://mlsanity.com/paper/2301.07227/) (2023)
- [Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence](https://mlsanity.com/paper/2608.12928/) (2026)

## Abstract (arXiv)

Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.