mlsanityData to 7 Oct 2026

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems

arXiv 2404.09486 ↗First posted 15 Apr 2024cs.CL cs.CV cs.SEcited by 38

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/likaixin2000/MMCode.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 40 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

25 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems21 Feb
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
10 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  7. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  8. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  9. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  10. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 22; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  4. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  5. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  6. 2025SlideCoder: Layout-aware RAG-enhanced Hierarchical Slide Generation from Designcited by 22
  7. 2025AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Paperscited by 18
  8. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 18
  9. 2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 15
  10. 2025Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generationcited by 14
  11. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  12. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
26 more
  1. 2025Charts Are Not Images: On the Challenges of Scientific Chart Editingcited by 9
  2. 2026EfficientPosterGen: Semantic-aware Efficient Poster Generation via Token Compression and Accurate Violation Detectioncited by 8
  3. 2025Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capabilitycited by 7
  4. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
  5. 2025MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoningcited by 6
  6. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 6
  7. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  8. 2024TurtleBench: A Visual Programming Benchmark in Turtle Geometrycited by 4
  9. 202590% Faster, 100% Code-Free: MLLM-Driven Zero-Code 3D Game Developmentcited by 4
  10. 2025OpusAnimation: Code-Based Dynamic Chart Generationcited by 3
  11. 2026Teaching Large Reasoning Models Effective Reflectioncited by 3
  12. 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
  13. 2026Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generationcited by 3
  14. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3
  15. 2026Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generationcited by 3
  16. 2025FairReason: Balancing Reasoning and Social Bias in MLLMscited by 2
  17. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  18. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critiquecited by 2
  19. 2026SVRepair: Structured Visual Reasoning for Automated Program Repaircited by 2
  20. 2026MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localizationcited by 2
  21. 2026VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?cited by 2
  22. 2025ChartAnchor: Chart Grounding with Structural-Semantic Fidelitycited by 1
  23. 2026Group Pattern Selection Optimization: Let LRMs Pick the Right Pattern for Reasoningcited by 1
  24. 2026TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphicscited by 0
  25. 2026Quantum Circuit Vision: Cost-Aware Evaluation of Visual AI Agents for Quantum Code Generationcited by 0
  26. 2026CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
  2. 2025Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
  3. 2024ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges
  4. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images
  5. 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
  6. 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems
  7. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
  8. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding