mlsanityData to 7 Oct 2026

What is Multimodality?

arXiv 2103.06304 ↗First posted 10 Mar 2021cs.AI cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the machine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important component of language grounding and a crucial milestone towards NLU.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

18 of 37 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  10. 2016Visual Dialogcited by 377
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2020Experience Grounds Languagecited by 241
6 more
  1. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  2. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  3. 2019Survey on Deep Neural Networks in Speech and Vision Systemscited by 37
  4. 2019Challenges and Prospects in Vision and Language Researchcited by 24
  5. 2019Multi-modality Latent Interaction Network for Visual Question Answeringcited by 23
  6. 2020Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challengecited by 7

Built on by

Not cited yet.