mlsanityData to 14 Oct 2026

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

arXiv 2311.17647 ↗First posted 29 Nov 2023cs.CV cs.AI cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent multimodal large language models (MLLMs) have shown promising instruction following capabilities on vision-language tasks. In this work, we introduce VISUAL MODALITY INSTRUCTION (VIM), and investigate how well multimodal models can understand textual instructions provided in pixels, despite not being explicitly trained on such data during pretraining or fine-tuning. We adapt VIM to eight benchmarks, including OKVQA, MM-Vet, MathVista, MMMU, and probe diverse MLLMs in both the text-modality instruction (TEM) setting and VIM setting. Notably, we observe a significant performance disparity between the original TEM and VIM settings for open-source MLLMs, indicating that open-source MLLMs face greater challenges when text instruction is presented solely in image form. To address this issue, we train v-MLLM, a generalizable model that is capable to conduct robust instruction following in both text-modality and visual-modality instructions.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

34 of 41 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  6. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
10 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2023Instruction Tuning with GPT-4cited by 686
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  6. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  7. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  8. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  9. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  10. 2022Language Modelling with Pixelscited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  2. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  3. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?cited by 8
  4. 2024BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Modelscited by 4
  5. 2026Same Answer, Different Representations: Hidden instability in VLMscited by 4
  6. 2026Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?cited by 1
  7. 2026When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
  2. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMs
  3. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  4. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  5. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
  8. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs