# Uni-SMART: Universal Science Multimodal Analysis and Research Transformer

arXiv: https://arxiv.org/abs/2403.10301 · HTML: https://mlsanity.com/paper/2403.10301/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables](https://mlsanity.com/paper/2105.06400/) (2021)
- [A framework for information extraction from tables in biomedical literature](https://mlsanity.com/paper/1902.10031/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools](https://mlsanity.com/paper/2506.20743/) (2025)
- [MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures](https://mlsanity.com/paper/2503.16096/) (2025)
- [Uni-Parser Technical Report](https://mlsanity.com/paper/2512.15098/) (2025)
- [MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures](https://mlsanity.com/paper/2603.28550/) (2026)
- [FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research](https://mlsanity.com/paper/2602.14285/) (2026)
- [Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System](https://mlsanity.com/paper/2603.12638/) (2026)
- [SciLENS: RL-Driven Autonomous Agents for Scientific Localized Evidence Navigation and Synthesis](https://mlsanity.com/paper/2609.03338/) (2026)

## Abstract (arXiv)

In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly articles, making in-depth literature analysis increasingly challenging and time-consuming. The emergence of Large Language Models (LLMs) has offered a new way to address this challenge. Known for their strong abilities in summarizing texts, LLMs are seen as a potential tool to improve the analysis of scientific literature. However, existing LLMs have their own limits. Scientific literature often includes a wide range of multimodal elements, such as tables, charts, and molecule, which are hard for text-focused LLMs to understand and analyze. This issue points to the urgent need for new solutions that can fully understand and analyze multimodal content in scientific literature. To answer this demand, we present Uni-SMART (Universal Science Multimodal Analysis and Research Transformer), an innovative model designed for in-depth understanding of multimodal scientific literature. Through rigorous quantitative evaluation across several domains, Uni-SMART demonstrates superior performance over other text-focused LLMs. Furthermore, our exploration extends to practical applications, including patent infringement detection and nuanced analysis of charts. These applications not only highlight Uni-SMART's adaptability but also its potential to revolutionize how we interact with scientific literature.