mlsanityData to 7 Oct 2026

m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks

arXiv 2403.11085 ↗First posted 17 Mar 2024cs.CV cs.CLcited by 38

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Real-world multi-modal problems are rarely solved by a single machine learning model, and often require multi-step computational plans that involve stitching several models. Tool-augmented LLMs hold tremendous promise for automating the generation of such computational plans. However, the lack of standardized benchmarks for evaluating LLMs as planners for multi-step multi-modal tasks has prevented a systematic study of planner design decisions. Should LLMs generate a full plan in a single shot or step-by-step? Should they invoke tools directly with Python code or through structured data formats like JSON? Does feedback improve planning? To answer these questions and more, we introduce m&m's: a benchmark containing 4K+ multi-step multi-modal tasks involving 33 tools that include multi-modal models, (free) public APIs, and image processing modules. For each of these task queries, we provide automatically generated plans using this realistic toolset. We further provide a high-quality subset of 1,565 task plans that are human-verified and correctly executable. With m&m's, we evaluate 10 popular LLMs with 2 planning strategies (multi-step vs. step-by-step planning), 2 plan formats (JSON vs. code), and 3 types of feedback (parsing/verification/execution). Finally, we summarize takeaways from our extensive experiments. Our dataset and code are available on HuggingFace (https://huggingface.co/datasets/zixianma/mnms) and Github (https://github.com/RAIVNLab/mnms).

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

35 of 52 references.

References · 35

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
23 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  7. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  10. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  11. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  12. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  13. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  14. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  15. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  16. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  17. 2019Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoningcited by 60
  18. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 47
  19. 2022Humans decompose tasks by trading off utility and computational costcited by 18
  20. 2023Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflowscited by 17
  21. 2023InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agentcited by 16
  22. 2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Usecited by 9
  23. 2023Agile Modeling: From Concept to Classifier in Minutescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 14; 2026: 13.

38 papers.

Sort:

By area:

  1. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  4. 2024GTA: A Benchmark for General Tool Agentscited by 77
  5. 2025ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Taskscited by 41
  6. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  7. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  8. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  9. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  10. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 22
  11. 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 21
  12. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 20
26 more
  1. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 20
  2. 2026ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruningcited by 18
  3. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 16
  4. 2025T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planningcited by 14
  5. 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13
  6. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 12
  7. 2024EcoAct: Economic Agent Determines When to Register What Actioncited by 11
  8. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  9. 2024CodeNav: Beyond tool-use to using real-world codebases with LLM agentscited by 8
  10. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  11. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?cited by 4
  12. 2026Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operationscited by 4
  13. 2025Sustainability assessment using multimodal AI agentscited by 3
  14. 2026From Tool Orchestration to Code Execution: A Study of MCP Design Choicescited by 3
  15. 2024SEAL: Suite for Evaluating API-use of LLMscited by 2
  16. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  17. 2025MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Modelscited by 2
  18. 2024Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Modelcited by 1
  19. 2026GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflowscited by 1
  20. 2025MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Usecited by 0
  21. 2026Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decompositioncited by 0
  22. 2026PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Searchcited by 0
  23. 2026TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Compositioncited by 0
  24. 2026AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculturecited by 0
  25. 2026Security and Privacy in Agentic AI: Grand Challenges and Future Directionscited by 0
  26. 2026Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Plancraft: an evaluation dataset for planning with LLM agents
  2. 2026TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents
  3. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models
  4. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios
  5. 2026PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents
  6. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
  7. 2025MMPlanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning
  8. 2026PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks