# OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery

arXiv: https://arxiv.org/abs/2501.15415 · HTML: https://mlsanity.com/paper/2501.15415/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding](https://mlsanity.com/paper/2311.11810/) (2023)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text](https://mlsanity.com/paper/2308.06911/) (2023)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning](https://mlsanity.com/paper/2402.17810/) (2024)
- [MolFM: A Multimodal Molecular Foundation Model](https://mlsanity.com/paper/2307.09484/) (2023)
- [MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation](https://mlsanity.com/paper/2205.14311/) (2022)
- [MolGrapher: Graph-based Visual Recognition of Chemical Structures](https://mlsanity.com/paper/2308.12234/) (2023)
- [Atom-Level Optical Chemical Structure Recognition with Limited Supervision](https://mlsanity.com/paper/2404.01743/) (2024)

## Built on by (most cited first, 8 of 8)

- [GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition](https://mlsanity.com/paper/2506.07553/) (2025)
- [BioMedGPT-Mol: Multi-task Learning for Molecular Understanding and Generation](https://mlsanity.com/paper/2512.04629/) (2025)
- [MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition](https://mlsanity.com/paper/2605.05832/) (2026)
- [MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition](https://mlsanity.com/paper/2608.03525/) (2026)
- [SubGrapher: Visual Fingerprinting of Chemical Structures](https://mlsanity.com/paper/2504.19695/) (2025)
- [DeepMoLM: Leveraging Visual and Geometric Structural Information for Molecule-Text Modeling](https://mlsanity.com/paper/2601.14732/) (2026)
- [COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training](https://mlsanity.com/paper/2604.23546/) (2026)
- [R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing](https://mlsanity.com/paper/2610.00700/) (2026)

## Abstract (arXiv)

Understanding the chemical structure from a graphical representation of a molecule is a challenging image caption task that would greatly benefit molecule-centric scientific discovery. Variations in molecular images and caption subtasks pose a significant challenge in both image representation learning and task modeling. Yet, existing methods only focus on a specific caption task that translates a molecular image into its graph structure, i.e., OCSR. In this paper, we propose the Optical Chemical Structure Understanding (OCSU) task, which extends low-level recognition to multilevel understanding and aims to translate chemical structure diagrams into readable strings for both machine and chemist. To facilitate the development of OCSU technology, we explore both OCSR-based and OCSR-free paradigms. We propose DoubleCheck to enhance OCSR performance via attentive feature enhancement for local ambiguous atoms. It can be cascaded with existing SMILES-based molecule understanding methods to achieve OCSU. Meanwhile, Mol-VL is a vision-language model end-to-end optimized for OCSU. We also construct Vis-CheBI20, the first large-scale OCSU dataset. Through comprehensive experiments, we demonstrate the proposed approaches excel at providing chemist-readable caption for chemical structure diagrams, which provide solid baselines for further research. Our code, model, and data are open-sourced at https://github.com/PharMolix/OCSU.