mlsanityData to 14 Oct 2026

Visual Programming: Compositional visual reasoning without training

arXiv 2211.11559 ↗First posted 18 Nov 2022cs.CV cs.AI cs.CLcited by 516

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VISPROG avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VISPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like VISPROG are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.

Key facts. Cited by 516 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 41 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

30 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
17 more
  1. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2022A Generalist Agentcited by 808
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  6. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  8. 2015Neural Module Networkscited by 376
  9. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  11. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  12. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  13. 2018DSFD: Dual Shot Face Detectorcited by 116
  14. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  15. 2018Explainable Neural Computation via Stack Neural Module Networkscited by 54
  16. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  17. 2022AugLy: Data Augmentations for Robustnesscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 112; 2024: 155; 2025: 124; 2026: 125.

516 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  4. 2023A Survey on Multimodal Large Language Modelscited by 909
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  6. 2023A Comprehensive Overview of Large Language Modelscited by 648
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  10. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  12. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
38 more
  1. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  2. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  3. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  6. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  9. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  11. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  12. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  13. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  14. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  15. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  16. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  17. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  18. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  19. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  20. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  21. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  22. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  23. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  24. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  25. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  26. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  27. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
  28. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  29. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  30. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 115
  31. 2023Language Prompt for Autonomous Drivingcited by 111
  32. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  33. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  34. 2024Large Multimodal Agents: A Surveycited by 105
  35. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  36. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  37. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  38. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91

50 most cited of 516 All 516 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  2. 2022A Benchmark for Compositional Visual Reasoning
  3. 2025NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning
  4. 2025VisPlay: Self-Evolving Vision-Language Models from Images
  5. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images
  6. 2026Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning
  7. 2023Learning Differentiable Logic Programs for Abstract Visual Reasoning
  8. 2019Learning by Abstraction: The Neural State Machine