mlsanityData to 7 Oct 2026

Large Language Models are Visual Reasoning Coordinators

arXiv 2310.15166 ↗First posted 23 Oct 2023cs.CV cs.CLcited by 58

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Visual reasoning requires multimodal perception and commonsense cognition of the world. Recently, multiple vision-language models (VLMs) have been proposed with excellent commonsense reasoning ability in various domains. However, how to harness the collective power of these complementary VLMs is rarely explored. Existing methods like ensemble still struggle to aggregate these models with the desired higher-order communications. In this work, we propose Cola, a novel paradigm that coordinates multiple VLMs for visual reasoning. Our key insight is that a large language model (LLM) can efficiently coordinate multiple VLMs by facilitating natural language communication that leverages their distinct and complementary capabilities. Extensive experiments demonstrate that our instruction tuning variant, Cola-FT, achieves state-of-the-art performance on visual question answering (VQA), outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Moreover, we show that our in-context learning variant, Cola-Zero, exhibits competitive performance in zero and few-shot settings, without finetuning. Through systematic ablation studies and visualizations, we validate that a coordinator LLM indeed comprehends the instruction prompts as well as the separate functionalities of VLMs; it then coordinates them to enable impressive visual reasoning capabilities.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 106 of its 125 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

106 of 125 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  2. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges26 Dec
  3. 2022Abstract Visual Reasoning with Tangram Shapes29 Nov
  4. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  5. 2022GAMR: A Guided Attention Model for (visual) Reasoning10 Jun
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr
  7. 2022A Review of Emerging Research Directions in Abstract Visual Reasoning21 Feb
  8. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices28 Jan

Other references · 98

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
38 more
  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  13. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  14. 2020Learning to summarize from human feedbackcited by 2,562
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  16. 2015VQA: Visual Question Answeringcited by 2,489
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  18. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  19. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  20. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  21. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  22. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  23. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  24. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  25. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  26. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  27. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  28. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  29. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  30. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  31. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  32. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  33. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  34. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  35. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  36. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  37. 2018CoQA: A Conversational Question Answering Challengecited by 674
  38. 2022Mixture-of-Experts with Expert Choice Routingcited by 611

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 20; 2025: 26; 2026: 6.

58 papers.

Sort:

By area:

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  2. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  3. 2024Large Multimodal Agents: A Surveycited by 99
  4. 2024Embodied Understanding of Driving Scenarioscited by 63
  5. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  6. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  7. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  8. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  9. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?cited by 36
  10. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  11. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  12. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
38 more
  1. 2024PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Modelcited by 20
  2. 2025Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detectioncited by 20
  3. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  4. 2024WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoningcited by 16
  5. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Modelscited by 13
  6. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  7. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  8. 2024Grounding Language Models for Visual Entity Recognitioncited by 12
  9. 2023Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answeringcited by 11
  10. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  11. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10
  12. 2024ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysiscited by 9
  13. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  14. 2025A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysiscited by 9
  15. 2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7
  16. 2025MetaWild: A Multimodal Dataset for Animal Re-Identification with Environmental Metadatacited by 6
  17. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5
  18. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?cited by 5
  19. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generationcited by 5
  20. 2025Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Testcited by 5
  21. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaborationcited by 4
  22. 2025QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safetycited by 4
  23. 2025Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observationscited by 4
  24. 2024Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQAcited by 3
  25. 2024Exploring Foundation Models in Remote Sensing Image Change Detection: A Comprehensive Surveycited by 3
  26. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  27. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  28. 2024Attribute-Aware Implicit Modality Alignment for Text Attribute Person Searchcited by 2
  29. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  30. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  31. 2025MCTS-EP: Empowering Embodied Planning with Online Preference Optimizationcited by 2
  32. 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimationcited by 2
  33. 2025LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matchingcited by 1
  34. 2025Clarification as Supervision: Reinforcement Learning for Vision-Language Interfacescited by 1
  35. 2026Netflix Artwork Personalization via LLM Post-trainingcited by 1
  36. 2026CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generationcited by 1
  37. 2025Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructionscited by 0
  38. 2025Vision language models have difficulty recognizing virtual objectscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models
  2. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  3. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  4. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
  5. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
  6. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  7. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  8. 2026Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models