mlsanityData to 14 Oct 2026

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

arXiv 2304.09842 ↗First posted 19 Apr 2023cs.CL cs.AI cs.CVcited by 356

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner. The project is available at https://chameleon-llm.github.io.

Key facts. Cited by 356 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 54 of its 72 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

54 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  2. 2022PAL: Program-aided Language Models18 Nov

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  13. 2023Instruction Tuning with GPT-4cited by 686
  14. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  15. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  16. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  17. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  18. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  19. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  20. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  21. 2015Neural Module Networkscited by 376
  22. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  23. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  24. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  25. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  26. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  27. 2018Bilinear Attention Networkscited by 280
  28. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  29. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  30. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  31. 2021Internet-Augmented Dialogue Generationcited by 217
  32. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  33. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  34. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  35. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  36. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  37. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  38. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 108; 2024: 120; 2025: 86; 2026: 42.

356 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  8. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  9. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  10. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  11. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  12. 2023Large Language Models as Tool Makerscited by 303
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  3. 2025Defeating Prompt Injections by Designcited by 297
  4. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  7. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  8. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  10. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  13. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  14. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  15. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  16. 2024Tool Learning with Large Language Models: A Surveycited by 223
  17. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  18. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  19. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 196
  20. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  21. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  22. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  23. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  24. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 168
  25. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  26. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  27. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  28. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  29. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  30. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  31. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  32. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  33. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  34. 2024Large Multimodal Agents: A Surveycited by 105
  35. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  36. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  37. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 96
  38. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96

50 most cited of 356 All 356 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios
  2. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  4. 2022Towards Reasoning in Large Language Models: A Survey
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2026Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis
  7. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  8. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models