mlsanityData to 14 Oct 2026

Grounding Language Models to Images for Multimodal Inputs and Outputs

arXiv 2301.13823 ↗First posted 31 Jan 2023cs.CL cs.AI cs.CVcited by 131

Not on the map. Why?

Cited by 51 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 69 of its 98 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

69 of 98 references.

References · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  14. 2022Emergent Abilities of Large Language Modelscited by 2,512
  15. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  16. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  18. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  19. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  20. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  21. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  22. 2016Generative Adversarial Text to Image Synthesiscited by 735
  23. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  24. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  26. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  27. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  28. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  29. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  30. 2016Visual Dialogcited by 377
  31. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  32. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  33. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  34. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  35. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  36. 2016Visual Storytellingcited by 176
  37. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  38. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 29; 2024: 60; 2025: 34; 2026: 8.

131 papers.

Sort:

By area:

  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  4. 2024The Platonic Representation Hypothesiscited by 382
  5. 2024What matters when building vision-language models?cited by 326
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2023Generating Images with Multimodal Language Modelscited by 260
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  10. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  11. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  12. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
38 more
  1. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  3. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  5. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  6. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  7. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  8. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  9. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  10. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  11. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  12. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  13. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
  14. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  15. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  16. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  17. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  18. 2023LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generationcited by 57
  19. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  20. 2025Harnessing the Universal Geometry of Embeddingscited by 55
  21. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  22. 2025Scaling Laws for Native Multimodal Modelscited by 50
  23. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  24. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  25. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  26. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  27. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  28. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  29. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 31
  30. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  31. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  32. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 24
  33. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 24
  34. 2023TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wildcited by 22
  35. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  36. 2023Context Diffusion: In-Context Aware Image Generationcited by 21
  37. 2024The Evolution of Multimodal Model Architecturescited by 21
  38. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21

50 most cited of 131 All 131 (JSON)