mlsanityData to 7 Oct 2026

Learning the Visualness of Text Using Large Vision-Language Models

arXiv 2305.10434 ↗First posted 11 May 2023cs.CL cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Visual text evokes an image in a person's mind, while non-visual text fails to do so. A method to automatically detect visualness in text will enable text-to-image retrieval and generation models to augment text with relevant images. This is particularly challenging with long-form text as text-to-image generation and retrieval models are often triggered for text that is designed to be explicitly visual in nature, whereas long-form text could contain many non-visual sentences. To this end, we curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. We also propose a fine-tuning strategy that adapts large vision-language models like CLIP by modifying the model's contrastive learning objective to map text identified as non-visual to a common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E. Project webpage: https://gaurav22verma.github.io/text-visualness/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 46 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

26 of 46 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
14 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  3. 2014How transferable are features in deep neural networks?cited by 1,216
  4. 2019Attention is not not Explanationcited by 629
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  6. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  7. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  8. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  9. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  10. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  11. 2016A Comprehensive Survey on Cross-modal Retrievalcited by 62
  12. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  13. 2022Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutionscited by 4
  14. 2021Multimodal and Contrastive Learning for Click Fraud Detectioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025See the Text: From Tokenization to Visual Reading
  2. 2026A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
  3. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluation
  4. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
  5. 2026Seeing Through Words: Controlling Visual Retrieval Quality with Language Models
  6. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  7. 2023Revisiting the Role of Language Priors in Vision-Language Models
  8. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding