mlsanityData to 14 Oct 2026

Probing the limitations of multimodal language models for chemistry and materials research

arXiv 2411.16955 ↗First posted 25 Nov 2024cs.LG cond-mat.mtrl-scicited by 32

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recent advancements in artificial intelligence have sparked interest in scientific assistants that could support researchers across the full spectrum of scientific workflows, from literature review to experimental design and data analysis. A key capability for such systems is the ability to process and reason about scientific information in both visual and textual forms - from interpreting spectroscopic data to understanding laboratory setups. Here, we introduce MaCBench, a comprehensive benchmark for evaluating how vision-language models handle real-world chemistry and materials science tasks across three core aspects: data extraction, experimental understanding, and results interpretation. Through a systematic evaluation of leading models, we find that while these systems show promising capabilities in basic perception tasks - achieving near-perfect performance in equipment identification and standardized data extraction - they exhibit fundamental limitations in spatial reasoning, cross-modal information synthesis, and multi-step logical inference. Our insights have important implications beyond chemistry and materials science, suggesting that developing reliable multimodal AI scientific assistants may require advances in curating suitable training data and approaches to training those models.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 96 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

55 of 96 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding6 Jul

Other references · 54

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2023Large Language Modelscited by 780
  10. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  12. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
38 more
  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  3. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  4. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  5. 2022Exploring Length Generalization in Large Language Modelscited by 206
  6. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  7. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  8. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  9. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  10. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  11. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  12. 2024ChemLLM: A Chemical Large Language Modelcited by 106
  13. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  14. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 93
  15. 2023Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineeringcited by 86
  16. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  17. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  18. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 61
  19. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patternscited by 57
  20. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 56
  21. 2023Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learningcited by 50
  22. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  23. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 48
  24. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  25. 2023Artificial intelligence and biological misuse: Differentiating risks of language models and biological design toolscited by 45
  26. 2024Prompt Design and Engineering: Introduction and Advanced Methodscited by 45
  27. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  28. 2024MassSpecGym: A benchmark for the discovery and identification of moleculescited by 38
  29. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  30. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  31. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  32. 2023Inductive reasoning in humans and large language modelscited by 24
  33. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  34. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 19
  35. 2022DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articlescited by 17
  36. 2024Benchmarking Large Language Models for Molecule Prediction Taskscited by 17
  37. 2024From Text to Insight: Large Language Models for Materials Science Data Extractioncited by 15
  38. 2024Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Modelscited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 16.

32 papers.

Sort:

By area:

  1. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  2. 2024Foundational Large Language Models for Materials Researchcited by 18
  3. 2025Towards LLM Agents for Earth Observationcited by 17
  4. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  5. 2025Multi-agent systems for chemical engineering: A review and perspectivecited by 9
  6. 2025UniFFBench: Evaluating Universal Machine Learning Force Fields Against Experimental Measurementscited by 6
  7. 2025SCI-Verifier: Scientific Verifier with Thinkingcited by 6
  8. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 6
  9. 2025AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materialscited by 5
  10. 2025ExpVid: A Benchmark for Experiment Video Understanding & Reasoningcited by 5
  11. 2024Autonomous Microscopy Experiments through Large Language Model Agentscited by 3
  12. 2025MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Modelscited by 3
20 more
  1. 2025Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectivescited by 2
  2. 2025L^2M^3OF: A Large Language Multimodal Model for Metal-Organic Frameworkscited by 2
  3. 2025SelfAI: A self-directed framework for long-horizon scientific discoverycited by 2
  4. 2026Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impactcited by 2
  5. 2026A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Eracited by 1
  6. 2026Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracycited by 1
  7. 2026ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answeringcited by 1
  8. 2026Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discoverycited by 1
  9. 2026OmniScientist: An Omni-Modal Omni-Discipline AI Scientistcited by 1
  10. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  11. 2026A collaborative agent with two lightweight synergistic models for autonomous crystal materials researchcited by 0
  12. 2026Large language model-enabled automated data extraction for concrete materials informaticscited by 0
  13. 2026MonitorVLM-v2: A Deployed Vision-Language Framework for Real-Time Safety Violation Detectioncited by 0
  14. 2026Strategy-first synthesis planning for complex natural productscited by 0
  15. 2026A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Imagescited by 0
  16. 2026Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responsescited by 0
  17. 2026XRF-to-Optical Field-of-View Localization with Vision Language Modelscited by 0
  18. 2026Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer depositioncited by 0
  19. 2026QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterizationcited by 0
  20. 2026ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
  2. 2025RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature
  3. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
  4. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistry
  5. 2026From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry
  6. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
  7. 2026MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science
  8. 2026ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding