# MolX: Enhancing Large Language Models for Molecular Understanding With A Multi-Modal Extension

arXiv: https://arxiv.org/abs/2406.06777 · HTML: https://mlsanity.com/paper/2406.06777/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [Unifying Molecular and Textual Representations via Multi-task Language Modelling](https://mlsanity.com/paper/2301.12586/) (2023)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning](https://mlsanity.com/paper/2402.17810/) (2024)
- [Root-aligned SMILES: A Tight Representation for Chemical Reaction Prediction](https://mlsanity.com/paper/2203.11444/) (2022)
- [Chemical-Reaction-Aware Molecule Representation Learning](https://mlsanity.com/paper/2109.09888/) (2021)
- [ReactionT5: a large-scale pre-trained model towards application of limited reaction data](https://mlsanity.com/paper/2311.06708/) (2023)
- [Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders](https://mlsanity.com/paper/1806.09300/) (2018)

## Built on by (most cited first, 5 of 5)

- [CodeGraph: Enhancing Graph Reasoning of LLMs with Code](https://mlsanity.com/paper/2408.13863/) (2024)
- [GraphSkill: Documentation-Guided Hierarchical Retrieval-Augmented Coding for Complex Graph Reasoning](https://mlsanity.com/paper/2603.06620/) (2026)
- [Chem4DLLM: 4D Multimodal LLMs for Chemical Dynamics Understanding](https://mlsanity.com/paper/2603.11924/) (2026)
- [Bolek: A Multimodal Language Model for Molecular Reasoning](https://mlsanity.com/paper/2605.02745/) (2026)
- [M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization](https://mlsanity.com/paper/2609.34491/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) with their strong task-handling capabilities have shown remarkable advancements across a spectrum of fields, moving beyond natural language understanding. However, their proficiency within the chemistry domain remains restricted, especially in solving molecule-related tasks. This challenge is attributed to their inherent limitations in comprehending molecules using only common textual representations, i.e. SMILES strings. In this study, we seek to enhance the ability of LLMs to comprehend molecules by equipping them with a multi-modal external module, termed MolX. Instead of directly using SMILES strings to represent a molecule, we utilize specific encoders to extract fine-grained features from both SMILES string and 2D molecular graph representations for feeding into an LLM. A hand-crafted molecular fingerprint is incorporated to leverage its embedded domain knowledge. To establish an alignment between MolX and the LLM's textual input space, the model in which the LLM is frozen, is pre-trained with a strategy including a diverse set of tasks. Experimental evaluations show that our proposed method outperforms baselines across downstream molecule-related tasks ranging from molecule-to-text translation to molecular property prediction, with and without fine-tuning the LLM, while only introducing a small number of trainable parameters-0.53% and 0.82%, respectively.