mlsanityData to 7 Oct 2026

ViperGPT: Visual Inference via Python Execution for Reasoning

arXiv 2303.08128 ↗First posted 14 Mar 2023cs.CVcited by 591

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.

Key facts. Cited by 591 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 79 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

58 of 79 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  2. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
38 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022PAL: Program-aided Language Modelscited by 635
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2015Neural Module Networkscited by 375
  8. 2022Visual Classification via Description from Large Language Modelscited by 331
  9. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  10. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  11. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  13. 2022Binding Language Models in Symbolic Languagescited by 224
  14. 2022TALM: Tool Augmented Language Modelscited by 197
  15. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  16. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  17. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  18. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  19. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  20. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  21. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  22. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  23. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  24. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  25. 2019Learning by Abstraction: The Neural State Machinecited by 113
  26. 2017The Consciousness Priorcited by 105
  27. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  28. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  29. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  30. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  31. 2022Video Graph Transformer for Video Question Answeringcited by 63
  32. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  33. 2019Language-Conditioned Graph Networks for Relational Reasoningcited by 55
  34. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  35. 2018Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Miningcited by 28
  36. 2022Open-vocabulary Attribute Detectioncited by 28
  37. 2021Coarse-to-Fine Reasoning for Visual Question Answeringcited by 25
  38. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 132; 2024: 165; 2025: 137; 2026: 157.

591 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  5. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  7. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  8. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  9. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  10. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  11. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  12. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  13. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  14. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  15. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  16. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  18. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  19. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  20. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  21. 2023Challenges and Applications of Large Language Modelscited by 251
  22. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  23. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  24. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  25. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  26. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  27. 2024Tool Learning with Large Language Models: A Surveycited by 221
  28. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  29. 2023Video Understanding with Large Language Models: A Surveycited by 203
  30. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  31. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  32. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  33. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  34. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  35. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  36. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  37. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  38. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156

50 most cited of 591 All 591 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VIPER Strike: Defeating Visual Reasoning CAPTCHAs via Structured Vision-Language Inference
  2. 2025RECODE: Reasoning Through Code Generation for Visual Question Answering
  3. 2026Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios
  4. 2026VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
  5. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
  6. 2025V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions
  7. 2025NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
  8. 2026ViThinker: Active Vision-Language Reasoning via Dynamic Perceptual Querying