mlsanityData to 14 Oct 2026

Learning More May Not Be Better: Knowledge Transferability in Vision and Language Tasks

arXiv 2208.10758 ↗First posted 23 Aug 2022cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Is more data always better to train vision-and-language models? We study knowledge transferability in multi-modal tasks. The current tendency in machine learning is to assume that by joining multiple datasets from different tasks their overall performance will improve. However, we show that not all the knowledge transfers well or has a positive impact on related tasks, even when they share a common goal. We conduct an exhaustive analysis based on hundreds of cross-experiments on 12 vision-and-language tasks categorized in 4 groups. Whereas tasks in the same group are prone to improve each other, results show that this is not always the case. Other factors such as dataset size or pre-training stage have also a great impact on how well the knowledge is transferred.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 23 Aug 2022.

Builds on

47 of 62 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
35 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  9. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  11. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  12. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  13. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  14. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  15. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  16. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  17. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  18. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  19. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  20. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  21. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  22. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 81
  23. 2018How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrievalcited by 74
  24. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  25. 2021Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Typescited by 54
  26. 2019Many Task Learning with Task Routingcited by 50
  27. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36
  28. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  29. 2022Multi-Attention Network for Compressed Video Referring Object Segmentationcited by 28
  30. 2018Multi-task Learning of Hierarchical Vision-Language Representationcited by 26
  31. 2019OmniNet: A unified architecture for multi-modal multi-task learningcited by 25
  32. 2015A Survey of Current Datasets for Vision and Language Researchcited by 13
  33. 2017Visual Question Answering as a Meta Learning Taskcited by 9
  34. 2021Transferring Domain-Agnostic Knowledge in Video Question Answeringcited by 7
  35. 2015How to reduce the number of rating scale items without predictability loss?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Understanding Task Transfer in Vision-Language Modelscited by 3
  2. 2025T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMscited by 1
  3. 2026Geometric Similarity in VLM Low-Level Vision Representationscited by 0