mlsanityData to 14 Oct 2026

Task Me Anything

arXiv 2406.11775 ↗First posted 17 Jun 2024cs.CV cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 17 Jun 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 5.

30 papers.

Sort:

By area:

  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  2. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 71
  3. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  4. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  5. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  6. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 25
  7. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 22
  8. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 21
  9. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  10. 2024LATTE: Learning to Think with Vision Specialistscited by 14
  11. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  12. 2025Automated Capability Discovery via Foundation Model Self-Explorationcited by 10
18 more
  1. 2024Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Modelscited by 9
  2. 2025Information Density Principle for MLLM Benchmarkscited by 9
  3. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scalecited by 8
  4. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generationcited by 6
  5. 2025Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Basecited by 6
  6. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
  7. 2024Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Trainingcited by 3
  8. 2025Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHubcited by 3
  9. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  10. 2026Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videoscited by 3
  11. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Eracited by 2
  12. 2026Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agentscited by 2
  13. 2024Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Modelcited by 1
  14. 2025LatentMove: Towards Complex Human Movement Video Generationcited by 1
  15. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
  16. 2026Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoningcited by 1
  17. 2026What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMscited by 0
  18. 2026Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  2. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  3. 2025Benchmarking Large and Small MLLMs
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  8. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation