Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities
Not on the map. Why?
Abstract (arXiv)
Contextualizing language technologies beyond a single language kindled embracing multiple modalities and languages. Individually, each of these directions undoubtedly proliferated into several NLP tasks. Despite this momentum, most of the multimodal research is primarily centered around English and multilingual research is primarily centered around contexts from text modality. Challenging this conventional setup, researchers studied the unification of multilingual and multimodal (MultiX) streams. The main goal of this work is to catalogue and characterize these works by charting out the categories of tasks, datasets and methods to address MultiX scenarios. To this end, we review the languages studied, gold or silver data with parallel annotations, and understand how these modalities and languages interact in modeling. We present an account of the modeling approaches along with their strengths and weaknesses to better understand what scenarios they can be used reliably. Following this, we present the high-level trends in the overall paradigm of the field. Finally, we conclude by presenting a road map of challenges and promising research directions.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 80 of its 140 references that are in the map. First posted on arXiv 30 Oct 2022.
Builds on
80 of 140 references.
References · 80
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
38 more
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
- 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
- 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
- 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
- 2018Constituency Parsing with a Self-Attentive Encodercited by 199
- 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
- 2021Visually Grounded Reasoning across Languages and Culturescited by 189
- 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
- 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
- 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
- 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
- 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
- 2015Order-Embeddings of Images and Languagecited by 113
- 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
- 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
- 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
- 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
- 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
- 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
- 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
- 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
- 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
- 2019NCLS: Neural Cross-Lingual Summarizationcited by 64
- 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
- 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
- 2019Visually Grounded Neural Syntax Acquisitioncited by 48
- 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
- 2021Grounding 'Grounding' in NLPcited by 44
- 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
- 2017Incorporating Global Visual Features into Attention-Based Neural Machine Translationcited by 39
50 most cited of 80 All 80 (JSON)
Built on by
Citing papers per year: 2023: 1.
1 paper.