mlsanityData to 7 Oct 2026

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

arXiv 2204.00598 ↗First posted 1 Apr 2022cs.CV cs.AI cs.CLcited by 513

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large pretrained (e.g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with external APIs and databases (e.g., web search), and (iii) robot perception and planning.

Key facts. Cited by 513 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 105 of its 165 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

105 of 165 references.

References · 105

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2017Mask R-CNNcited by 7,259
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2020Rethinking Attention with Performerscited by 1,584
  13. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  14. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  15. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  16. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  17. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  19. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  20. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  21. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  22. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  23. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  24. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  25. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  26. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  27. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  28. 2021Robust fine-tuning of zero-shot modelscited by 785
  29. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  30. 2022Language-driven Semantic Segmentationcited by 711
  31. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  32. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  33. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  34. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  35. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  36. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  37. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  38. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 56; 2023: 184; 2024: 146; 2025: 86; 2026: 41.

513 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
38 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  11. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  12. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  13. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  14. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  15. 2023Cognitive Architectures for Language Agentscited by 422
  16. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  17. 2023Augmented Language Models: a Surveycited by 398
  18. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  19. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  20. 2023Language Models can Solve Computer Taskscited by 371
  21. 2022Visual Language Maps for Robot Navigationcited by 362
  22. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  23. 2022Visual Classification via Description from Large Language Modelscited by 331
  24. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  25. 2023Tool Learning with Foundation Modelscited by 302
  26. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  27. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  28. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  29. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  30. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  31. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  32. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  35. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  36. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  37. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  38. 2022Learning Video Representations from Large Language Modelscited by 204

50 most cited of 513 All 513 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  2. 2023On the Performance of Multimodal Language Models
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2025Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild
  5. 2025VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs
  6. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learning
  7. 2023Language Is Not All You Need: Aligning Perception with Language Models
  8. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning