# MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

arXiv: https://arxiv.org/abs/2603.28550 · HTML: https://mlsanity.com/paper/2603.28550/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation](https://mlsanity.com/paper/2205.14311/) (2022)
- [MolGrapher: Graph-based Visual Recognition of Chemical Structures](https://mlsanity.com/paper/2308.12234/) (2023)
- [OpenChemIE: An Information Extraction Toolkit For Chemistry Literature](https://mlsanity.com/paper/2404.01462/) (2024)
- [MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild](https://mlsanity.com/paper/2411.11098/) (2024)
- [MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognition](https://mlsanity.com/paper/2403.03691/) (2024)
- [Improving Chemical Named Entity Recognition in Patents with Contextualized Word Embeddings](https://mlsanity.com/paper/1907.02679/) (2019)
- [Uni-SMART: Universal Science Multimodal Analysis and Research Transformer](https://mlsanity.com/paper/2403.10301/) (2024)
- [MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures](https://mlsanity.com/paper/2503.16096/) (2025)
- [Intelligent System for Automated Molecular Patent Infringement Assessment](https://mlsanity.com/paper/2412.07819/) (2024)

## Built on by (most cited first, 2 of 2)

- [MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition](https://mlsanity.com/paper/2607.28532/) (2026)
- [R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing](https://mlsanity.com/paper/2610.00700/) (2026)

## Abstract (arXiv)

Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets are released publicly.