mlsanityData to 14 Oct 2026

Modular Visual Question Answering via Code Generation

arXiv 2306.05392 ↗First posted 8 Jun 2023cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3% and on the GQA dataset by roughly 2% compared to the few-shot baseline that does not employ code generation.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

32 of 42 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
20 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  6. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  9. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  10. 2022Binding Language Models in Symbolic Languagescited by 228
  11. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  12. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  13. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  14. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  15. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  16. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  17. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  18. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  19. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  20. 2020Obtaining Faithful Interpretations from Compositional Neural Networkscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 26; 2025: 8; 2026: 7.

48 papers.

Sort:

By area:

  1. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  3. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  4. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 75
  5. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  6. 2024Real2Code: Reconstruct Articulated Objects via Code Generationcited by 69
  7. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 68
  8. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  9. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  10. 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 49
  11. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
  12. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
36 more
  1. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  2. 2024FlowMind: Automatic Workflow Generation with LLMscited by 30
  3. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25
  4. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 20
  5. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  6. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesiscited by 17
  7. 2023API-Assisted Code Generation for Question Answering on Varied Table Structurescited by 14
  8. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  9. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 14
  10. 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcementcited by 12
  11. 2024Grounding Language in Multi-Perspective Referential Communicationcited by 12
  12. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  13. 2025CountGD++: Generalized Prompting for Open-World Countingcited by 8
  14. 2023Analyzing Modular Approaches for Visual Question Decompositioncited by 6
  15. 2023Recursive Visual Programmingcited by 6
  16. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  17. 2025Synthesizing Visual Concepts as Vision-Language Programscited by 6
  18. 2024FISHNET: Financial Intelligence from Sub-querying, Harmonizing, Neural-Conditioning, Expert Swarms, and Task Planningcited by 5
  19. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  20. 2024PropTest: Automatic Property Testing for Improved Visual Programmingcited by 4
  21. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understandingcited by 4
  22. 2026Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generationcited by 4
  23. 2024Visual Program Distillation with Template-Based Augmentationcited by 3
  24. 2025Understanding Complexity in VideoQA via Visual Program Generationcited by 3
  25. 2024Automatic benchmarking of large multimodal models via iterative experiment programmingcited by 2
  26. 2025Resource-efficient Inference with Foundation Model Programscited by 2
  27. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  28. 2024LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classificationcited by 1
  29. 2024VQA Training Sets are Self-play Environments for Generating Few-shot Poolscited by 1
  30. 2024AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answeringcited by 1
  31. 2024Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answeringcited by 1
  32. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  33. 2026Reasoning with Pixel-level Precision: QVLM Architecture and SQuID Dataset for Quantitative Geospatial Analyticscited by 1
  34. 2026Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answeringcited by 0
  35. 2026VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinementcited by 0
  36. 2026Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answeringcited by 0