mlsanityData to 7 Oct 2026

VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

arXiv 2402.07398 ↗First posted 12 Feb 2024cs.AIcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper presents VisLingInstruct, a novel approach to advancing Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. VisLingInstruct tackles this by autonomously evaluating and optimizing instructional texts through In-Context Learning, improving the synergy between visual perception and linguistic expression in MMLMs. Alongside this instructional advancement, we have also optimized the visual feature extraction modules in MMLMs, further augmenting their responsiveness to textual content. Our comprehensive experiments on MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly improves zero-shot performance in visual multi-modal tasks. Notably, it achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on the TextVQA and HatefulMemes datasets. Our main code is available at https://github.com/Zhudongsheng75/VisLingInstruct.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 44 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

29 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
16 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  8. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2016Visual Dialogcited by 377
  11. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  12. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  13. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  14. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  15. 2022What Makes Pre-trained Language Models Better Zero-shot Learners?cited by 23
  16. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  2. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2024The Evolution of Multimodal Model Architecturescited by 21
  5. 2024Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflectioncited by 18
  6. 2025Divide-Then-Aggregate: An Efficient Tool Learning Method via Parallel Tool Invocationcited by 16
  7. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  8. 2025What Makes a Good Natural Language Prompt?cited by 6
  9. 2024Smart Vision-Language Reasonerscited by 5
  10. 2025Large Language Models for Zero-Shot Multicultural Name Recognitioncited by 1
  11. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generationcited by 1
  12. 2025LVLM-Composer's Explicit Planning for Image Generationcited by 0
4 more
  1. 2025VisuCraft: Enhancing Large Vision-Language Models for Complex Visual-Guided Creative Content Generation via Structured Information Extractioncited by 0
  2. 2025MM-FusionNet: Context-Aware Dynamic Fusion for Multi-modal Fake News Detection with Large Vision-Language Modelscited by 0
  3. 2025ContextGuard-LVLM: Enhancing News Veracity through Fine-grained Cross-modal Contextual Consistency Verificationcited by 0
  4. 2026An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2023On the Performance of Multimodal Language Models
  4. 2024Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
  5. 2023VILA: On Pre-training for Visual Language Models
  6. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
  7. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  8. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?