mlsanityData to 7 Oct 2026

ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area

arXiv 2408.07246 ↗First posted 14 Aug 2024cs.LG cs.CVcited by 68

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success and have been applied across various scientific fields, including chemistry. However, many chemical tasks require the processing of visual information, which cannot be successfully handled by existing chemical LLMs. This brings a growing need for models capable of integrating multimodal information in the chemical domain. In this paper, we introduce ChemVLM, an open-source chemical multimodal large language model specifically designed for chemical applications. ChemVLM is trained on a carefully curated bilingual multimodal dataset that enhances its ability to understand both textual and visual chemical information, including molecular structures, reactions, and chemistry examination questions. We develop three datasets for comprehensive evaluation, tailored to Chemical Optical Character Recognition (OCR), Multimodal Chemical Reasoning (MMCR), and Multimodal Molecule Understanding tasks. We benchmark ChemVLM against a range of open-source and proprietary multimodal large language models on various tasks. Experimental results demonstrate that ChemVLM achieves competitive performance across all evaluated tasks. Our model can be found at https://huggingface.co/AI4Chem/ChemVLM-26B.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 42 references that are in the map. First posted on arXiv 14 Aug 2024.

Builds on

21 of 42 references.

References · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
9 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  3. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  4. 2022MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generationcited by 48
  5. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  6. 2018Molecular Structure Extraction From Documents Using Deep Learningcited by 25
  7. 2020ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learningcited by 18
  8. 2023RxnScribe: A Sequence Generation Model for Reaction Diagram Parsingcited by 16
  9. 2024PEaCE: A Chemistry-Oriented Dataset for Optical Character Recognition on Scientific Documentscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 26; 2026: 35.

68 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  3. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  4. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  5. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  6. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  7. 2024Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generationcited by 22
  8. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 20
  9. 2025ChemMLLM: Chemical Multimodal Large Language Modelcited by 16
  10. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  11. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  12. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
38 more
  1. 2026GENIUS: Generative Fluid Intelligence Evaluation Suitecited by 11
  2. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 10
  3. 2024HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embeddingcited by 10
  4. 2025MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMscited by 9
  5. 2025Exploiting Edited Large Language Models as General Scientific Optimizerscited by 7
  6. 2025GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognitioncited by 7
  7. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  8. 2026LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoningcited by 6
  9. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  10. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  11. 2025TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Taskscited by 5
  12. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 5
  13. 2026El Agente Estructural: An Artificially Intelligent Molecular Editorcited by 5
  14. 2025MAPS: Multi-Agent Personality Shaping for Collaborative Reasoningcited by 4
  15. 2025CoMemo: LVLMs Need Image Context with Image Memorycited by 4
  16. 2025Stress-Testing Multimodal Foundation Models for Crystallographic Reasoningcited by 4
  17. 2026SIQA: Toward Reliable Scientific Image Quality Assessmentcited by 4
  18. 2025AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Usecited by 3
  19. 2025CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captionscited by 3
  20. 2025BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Modelscited by 3
  21. 2026PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Taskscited by 3
  22. 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
  23. 2026G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Modelscited by 3
  24. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3
  25. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimationcited by 2
  26. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  27. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMscited by 2
  28. 2025ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judgescited by 1
  29. 2025Large Language Models Transform Organic Synthesis From Reaction Prediction to Automationcited by 1
  30. 2026ForgetMark: Stealthy Fingerprint Embedding via Targeted Unlearning in Language Modelscited by 1
  31. 2026MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognitioncited by 1
  32. 2026Towards Characterizing Scientific Image Utility and Upgradabilitycited by 1
  33. 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Trainingcited by 1
  34. 2026Learning Mechanistic Reasoning for Chemical Reactions with Large Language Modelscited by 1
  35. 2026ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMscited by 1
  36. 2026Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognitioncited by 1
  37. 2025Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performancecited by 0
  38. 2025Language-Native Materials Processing Design by Lightly Structured Text Database and Reasoning Large Language Modelcited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
  2. 2026ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding
  3. 2025RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature
  4. 2025A Multi-Agent System Enables Versatile Information Extraction from the Chemical Literature
  5. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistry
  6. 2024Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation
  7. 2026Molecular Representations for Large Language Models
  8. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistry