mlsanityData to 7 Oct 2026

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning

arXiv 2112.05253 ↗First posted 9 Dec 2021cs.CV cs.CLcited by 85

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying optimization compared to previous approaches. Importantly, the language model weights remain unchanged during training, allowing for transfer of encyclopedic knowledge and in-context learning abilities from language pretraining. MAGMA outperforms Frozen on open-ended generative tasks, achieving state of the art results on the OKVQA benchmark and competitive results on a range of other popular VL benchmarks, while pretraining on 0.2% of the number of samples used to train SimVLM.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

50 of 64 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  19. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  20. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  21. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  22. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  23. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  24. 2018nocaps: novel object captioning at scalecited by 444
  25. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  26. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  27. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  28. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  29. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  30. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  31. 2019Connecting Vision and Language with Localized Narrativescited by 225
  32. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  33. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  34. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  35. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36
  36. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30
  37. 2021Coarse-to-Fine Reasoning for Visual Question Answeringcited by 25
  38. 2021Multimodal Conditionality for Natural Language Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 38; 2024: 21; 2025: 6; 2026: 3.

85 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  4. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  5. 2023Generating Images with Multimodal Language Modelscited by 259
  6. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  7. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  8. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  9. 2022Linearly Mapping from Image to Text Spacecited by 139
  10. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  11. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  12. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 63
38 more
  1. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  2. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  3. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  4. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  5. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  6. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  7. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  8. 2023Quality-Diversity through AI Feedbackcited by 40
  9. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  10. 2024A Vision Check-up for Language Modelscited by 36
  11. 2024Private Attribute Inference from Images with Vision-Language Modelscited by 35
  12. 2023Visual Tuningcited by 34
  13. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  14. 2023AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulationcited by 30
  15. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  16. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  17. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  18. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  19. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  20. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  21. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  22. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  23. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  24. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
  25. 2022Does CLIP Know My Face?cited by 19
  26. 2023Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!cited by 19
  27. 2024COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Trainingcited by 18
  28. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  29. 2023Learning by Self-Explainingcited by 17
  30. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  31. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  32. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 14
  33. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  34. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  35. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Modelscited by 12
  36. 2024VIXEN: Visual Text Comparison Network for Image Difference Captioningcited by 12
  37. 2025CTRL-O: Language-Controllable Object-Centric Visual Representation Learningcited by 12
  38. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11

50 most cited of 85 All 85 (JSON)