mlsanityData to 7 Oct 2026

VLIS: Unimodal Language Models Guide Multimodal Language Generation

arXiv 2310.09767 ↗First posted 15 Oct 2023cs.CL cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field. However, existing vision-language models face challenges in tasks that require complex linguistic understanding. To address this issue, we introduce Visual-Language models as Importance Sampling weights (VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves vision-language models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 80 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

67 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation12 Mar
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2018Hierarchical Neural Story Generationcited by 1,182
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  17. 2018Object Hallucination in Image Captioningcited by 661
  18. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  19. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  24. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  25. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  26. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  27. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  28. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  29. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  30. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  31. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  32. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  33. 2022A Contrastive Framework for Neural Text Generationcited by 224
  34. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  35. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  36. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  37. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  38. 2018Learning to Write with Cooperative Discriminatorscited by 119

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024What Makes for Good Image Captions?cited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  3. 2024An Introduction to Vision-Language Modeling
  4. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  5. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  6. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  7. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  8. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models