mlsanityData to 7 Oct 2026

HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks

arXiv 2410.12381 ↗First posted 16 Oct 2024cs.CV cs.AIcited by 19

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Understanding and reasoning over diagrams is a fundamental aspect of human intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive capabilities across various tasks, existing benchmarks lack comprehensive evaluation of their diagram interpretation and reasoning abilities, particularly in coding contexts. We present HumanEval-V, a rigorous benchmark of human-annotated coding tasks that spans six task types and evaluates diverse visual reasoning capabilities. Each task features carefully crafted diagrams paired with function signatures and test cases, employing novel code generation tasks to thoroughly assess models' diagram comprehension. Through extensive experiments with 22 LMMs, we find that even top-performing models achieve modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1, highlighting substantial room for improvement. Our analysis reveals that current LMMs struggle with spatial transformations, topological relationships, and dynamic patterns that humans find intuitive. These findings provide valuable insights for advancing LMMs' visual reasoning abilities. We have open-sourced our code and benchmark at https://github.com/HumanEval-V/HumanEval-V-Benchmark.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 55 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

32 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation14 Jun
  2. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  3. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning21 Apr
  4. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
12 more
  1. 2022CodeT: Code Generation with Generated Testscited by 369
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  3. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  4. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  5. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  6. 2024Pixtral 12Bcited by 174
  7. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  8. 2019On the Measure of Intelligencecited by 129
  9. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  10. 2023Modular Visual Question Answering via Code Generationcited by 47
  11. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  12. 2024Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plotscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  3. 2025Large language models for automated scholarly paper review: A surveycited by 59
  4. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  5. 2026STEP3-VL-10B Technical Reportcited by 28
  6. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  8. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  9. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  10. 2026MDGYM: Benchmarking AI Agents on Molecular Simulationscited by 3
  11. 2025MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Modelscited by 2
  12. 2025Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Frameworkcited by 1
7 more
  1. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  2. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  3. 2026TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoningcited by 1
  4. 2026Evaluating Large Language Models for Security Bug Report Predictioncited by 1
  5. 2026Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensemblescited by 1
  6. 2026ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generationcited by 0
  7. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity
  2. 2025Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark
  3. 2025Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning
  4. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought
  5. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles
  6. 2026CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions
  7. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
  8. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning