mlsanityData to 14 Oct 2026

ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

arXiv 2406.09961 ↗First posted 14 Jun 2024cs.SE cs.CL cs.CVcited by 129

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. ChartMimic includes 4,800 human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span 18 regular types and 4 advanced types, diversifying into 201 subcategories. Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of 3 proprietary models and 14 open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of 82.2 and 61.6, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 51 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

33 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts26 Dec
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
17 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  4. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  5. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  6. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  7. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  8. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  9. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  10. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  11. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 102
  12. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  13. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  14. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 40
  15. 2023ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summariescited by 32
  16. 2021Understanding Data Visualization Design Practicecited by 5
  17. 2024A New Era in Human Factors Engineering: A Survey of the Applications and Prospects of Large Multimodal Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 50; 2026: 69.

129 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  5. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  6. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 92
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  8. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  9. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 65
  10. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  11. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  12. 2024Can LLMs Understand Time Series Anomalies?cited by 41
38 more
  1. 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 34
  2. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 32
  3. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedbackcited by 32
  4. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  5. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  6. 2026PaperBanana: Automating Academic Illustration for AI Scientistscited by 26
  7. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 25
  8. 2025MorphMark: Flexible Adaptive Watermarking for Large Language Modelscited by 24
  9. 2024Drawing Pandas: A Benchmark for LLMs in Generating Plotting Codecited by 23
  10. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 21
  11. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 20
  12. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  13. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 19
  14. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  15. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 16
  16. 2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 16
  17. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
  18. 2025VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representationcited by 16
  19. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  20. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  21. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generationcited by 14
  22. 2026Multimodal OCR: Parse Anything from Documentscited by 14
  23. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  24. 2025Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilitiescited by 13
  25. 2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answeringcited by 13
  26. 2026Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationcited by 13
  27. 2026OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Modelscited by 12
  28. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  29. 2025ChartEditor: A Reinforcement Learning Framework for Robust Chart Editingcited by 11
  30. 2025Charts Are Not Images: On the Challenges of Scientific Chart Editingcited by 11
  31. 2025Improved Iterative Refinement for Chart-to-Code Generation via Structured Instructioncited by 9
  32. 2025ChartM3: Benchmarking Chart Editing with Multimodal Instructionscited by 9
  33. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 9
  34. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 9
  35. 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 8
  36. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 8
  37. 2025Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Modelscited by 7
  38. 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 7

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Charts to Code: A Hierarchical Benchmark for Multimodal Models
  2. 2026ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation
  3. 2025ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension
  4. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
  5. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
  6. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
  7. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
  8. 2026CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution