mlsanityData to 14 Oct 2026

Linearly Mapping from Image to Text Space

arXiv 2209.15162 ↗First posted 30 Sep 2022cs.CL cs.LGcited by 142

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber

Key facts. Cited by 142 papers in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

35 of 49 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 804
23 more
  1. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  2. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  3. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  4. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2018nocaps: novel object captioning at scalecited by 444
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  8. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  11. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  12. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  13. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  14. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  15. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  16. 2022Language Models are General-Purpose Interfacescited by 97
  17. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  18. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  19. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  20. 2022Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioningcited by 21
  21. 2015Language Models for Image Captioning: The Quirks and What Workscited by 19
  22. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  23. 2022Training Vision-Language Transformers from Captionscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 32; 2024: 46; 2025: 33; 2026: 29.

142 papers.

Sort:

By area:

  1. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  2. 2024The Platonic Representation Hypothesiscited by 382
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  5. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  7. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  8. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  9. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  10. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 119
  11. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  12. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
38 more
  1. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  2. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 64
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2025Scaling Laws for Native Multimodal Modelscited by 50
  5. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  6. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 49
  7. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 45
  8. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  9. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  10. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  11. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  12. 2024A Vision Check-up for Language Modelscited by 36
  13. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 36
  14. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  15. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 34
  16. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  17. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  18. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  19. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  20. 2024GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discoverycited by 27
  21. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 27
  22. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  23. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutationscited by 25
  24. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 25
  25. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 25
  26. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 24
  27. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  28. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  29. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  30. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  31. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  32. 2024Language Representations Can be What Recommenders Need: Findings and Potentialscited by 18
  33. 2024Objective drives the consistency of representational similarity across datasetscited by 18
  34. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 17
  35. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  36. 2025With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide Youcited by 16
  37. 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 16
  38. 2024What Do Language Models Hear? Probing for Auditory Representations in Language Modelscited by 15

50 most cited of 142 All 142 (JSON)