# Image and Data Mining in Reticular Chemistry Using GPT-4V arXiv: https://arxiv.org/abs/2312.05468 ยท HTML: https://mlsanity.com/paper/2312.05468/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 60 references that are in the map. - First posted on arXiv 9 Dec 2023. ## Builds on (most cited first, 21 of 21) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023) - [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023) - [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023) - [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023) - [14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon](https://mlsanity.com/paper/2306.06283/) (2023) - [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023) - [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023) - [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023) - [ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis](https://mlsanity.com/paper/2306.11296/) (2023) - [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023) - [Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V](https://mlsanity.com/paper/2310.19061/) (2023) - [ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback](https://mlsanity.com/paper/2305.18090/) (2023) - [Using Machine Learning and Data Mining to Leverage Community Knowledge for the Engineering of Stable Metal-Organic Frameworks](https://mlsanity.com/paper/2106.13327/) (2021) - [ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworks](https://mlsanity.com/paper/2308.01423/) (2023) - [Revisiting Prompt Engineering via Declarative Crowdsourcing](https://mlsanity.com/paper/2308.03854/) (2023) - [MOFSimplify: Machine Learning Models with Extracted Stability Data of Three Thousand Metal-Organic Frameworks](https://mlsanity.com/paper/2109.08098/) (2021) - [A Database of Ultrastable MOFs Reassembled from Stable Fragments with Machine Learning Models](https://mlsanity.com/paper/2210.14191/) (2022) ## Built on by (most cited first, 6 of 6) - [El Agente: An Autonomous Agent for Quantum Chemistry](https://mlsanity.com/paper/2505.02484/) (2025) - [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024) - [Materials science in the era of large language models: a perspective](https://mlsanity.com/paper/2403.06949/) (2024) - [Agent-based Learning of Materials Datasets from Scientific Literature](https://mlsanity.com/paper/2312.11690/) (2023) - [Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots](https://mlsanity.com/paper/2503.12326/) (2025) - [The Rise of Generative AI for Metal-Organic Framework Design and Synthesis](https://mlsanity.com/paper/2508.13197/) (2025) ## Abstract (arXiv) The integration of artificial intelligence into scientific research has reached a new pinnacle with GPT-4V, a large language model featuring enhanced vision capabilities, accessible through ChatGPT or an API. This study demonstrates the remarkable ability of GPT-4V to navigate and obtain complex data for metal-organic frameworks, especially from graphical sources. Our approach involved an automated process of converting 346 scholarly articles into 6240 images, which represents a benchmark dataset in this task, followed by deploying GPT-4V to categorize and analyze these images using natural language prompts. This methodology enabled GPT-4V to accurately identify and interpret key plots integral to MOF characterization, such as nitrogen isotherms, PXRD patterns, and TGA curves, among others, with accuracy and recall above 93%. The model's proficiency in extracting critical information from these plots not only underscores its capability in data mining but also highlights its potential in aiding the creation of comprehensive digital databases for reticular chemistry. In addition, the extracted nitrogen isotherm data from the selected literature allowed for a comparison between theoretical and experimental porosity values for over 200 compounds, highlighting certain discrepancies and underscoring the importance of integrating computational and experimental data. This work highlights the potential of AI in accelerating scientific discovery and innovation, bridging the gap between computational tools and experimental research, and paving the way for more efficient, inclusive, and comprehensive scientific inquiry.