mlsanityData to 7 Oct 2026

A Vision Check-up for Language Models

arXiv 2401.01862 ↗First posted 3 Jan 2024cs.CV cs.CL cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

What does learning to model relationships between strings teach large language models (LLMs) about the visual world? We systematically evaluate LLMs' abilities to generate and recognize an assortment of visual concepts of increasing complexity and then demonstrate how a preliminary visual representation learning system can be trained using models of text. As language models lack the ability to consume or output visual information as pixels, we use code to represent images in our study. Although LLM-generated images do not look like natural images, results on image generation and the ability of models to correct these generated images indicate that precise modeling of strings can teach language models about numerous aspects of the visual world. Furthermore, experiments on self-supervised visual representation learning, utilizing images generated with text models, highlight the potential to train vision models capable of making semantic assessments of natural images using just LLMs.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

38 of 56 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
26 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  2. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  3. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  7. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  8. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  9. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  10. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  11. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  12. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  13. 2022Linearly Mapping from Image to Text Spacecited by 139
  14. 2018Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approachcited by 119
  15. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  16. 2021Pre-training without Natural Imagescited by 100
  17. 2021Learning to See by Looking at Noisecited by 92
  18. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  19. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  20. 2022Procedural Image Programs for Representation Learningcited by 28
  21. 2023Dissociating language and thought in large language modelscited by 27
  22. 2021Improving Fractal Pre-trainingcited by 26
  23. 2020Constructing Taxonomies from Pretrained Language Modelscited by 21
  24. 2023Visual Atoms: Pre-training Vision Transformers with Sinusoidal Wavescited by 20
  25. 2014On Rendering Synthetic Images for Training an Object Detectorcited by 18
  26. 2022Do language models have coherent mental models of everyday things?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 17; 2026: 5.

36 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 362
  2. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 63
  3. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patternscited by 54
  4. 2024CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solverscited by 40
  5. 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 37
  6. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  7. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  8. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  9. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  10. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  11. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  12. 2025WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agentscited by 21
24 more
  1. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 21
  2. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  3. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  4. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMscited by 14
  5. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  6. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  7. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  8. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  9. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generationcited by 5
  10. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  11. 2026The Geometry of Representational Failures in Vision Language Modelscited by 5
  12. 2025A Neurosymbolic Agent System for Compositional Visual Reasoningcited by 4
  13. 2026The Indra Representation Hypothesis for Multimodal Alignmentcited by 4
  14. 2024Better Language Models Exhibit Higher Visual Alignmentcited by 2
  15. 2025MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Modelscited by 1
  16. 2025Probing Audio-Generation Capabilities of Text-Based Language Modelscited by 1
  17. 2024Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorderscited by 0
  18. 2025Generating Animated Layouts as Structured Text Representationscited by 0
  19. 2025Large Language Models' Internal Perception of Symbolic Musiccited by 0
  20. 2025MissionHD: Hyperdimensional Refinement of Distribution-Deficient Reasoning Graphs for Video Anomaly Detectioncited by 0
  21. 2025An Exploratory Study on Abstract Images and Visual Representations Learned from Themcited by 0
  22. 2026ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Modelscited by 0
  23. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Modelscited by 0
  24. 2026FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videoscited by 0