# Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration

arXiv: https://arxiv.org/abs/2601.12256 · HTML: https://mlsanity.com/paper/2601.12256/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders](https://mlsanity.com/paper/2408.15998/) (2024)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://mlsanity.com/paper/2311.16208/) (2023)
- [MoVA: Adapting Mixture of Vision Experts to Multimodal Context](https://mlsanity.com/paper/2404.13046/) (2024)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO](https://mlsanity.com/paper/2506.07464/) (2025)
- [BERN2: an advanced neural biomedical named entity recognition and normalization tool](https://mlsanity.com/paper/2201.02080/) (2022)
- [TableRAG: Million-Token Table Understanding with Language Models](https://mlsanity.com/paper/2410.04739/) (2024)
- [Self-positioning Point-based Transformer for Point Cloud Understanding](https://mlsanity.com/paper/2303.16450/) (2023)
- [LLaMo: Large Language Model-based Molecular Graph Assistant](https://mlsanity.com/paper/2411.00871/) (2024)
- [PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes](https://mlsanity.com/paper/2406.13193/) (2024)
- [PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs](https://mlsanity.com/paper/2510.25808/) (2025)
- [TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing](https://mlsanity.com/paper/2511.13283/) (2025)
- [Vision Language Model is NOT All You Need: Augmentation Strategies for Molecule Language Models](https://mlsanity.com/paper/2407.09043/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated their instruction-following capabilities and achieved powerful performance on various tasks. Inspired by their success, recent works in the molecular domain have led to the development of large molecular language models (LMLMs) that integrate 1D molecular strings or 2D molecular graphs into the language models. However, existing LMLMs often suffer from hallucination and limited robustness, largely due to inadequate integration of diverse molecular modalities such as 1D sequences, 2D molecular graphs, and 3D conformations. To address these limitations, we propose CoLLaMo, a large language model-based molecular assistant equipped with a multi-level molecular modality-collaborative projector. The relation-aware modality-collaborative attention mechanism in the projector facilitates fine-grained and relation-guided information exchange between atoms by incorporating 2D structural and 3D spatial relations. Furthermore, we present a molecule-centric new automatic measurement, including a hallucination assessment metric and GPT-based caption quality evaluation to address the limitations of token-based generic evaluation metrics (i.e., BLEU) widely used in assessing molecular comprehension of LMLMs. Our extensive experiments demonstrate that our CoLLaMo enhances the molecular modality generalization capabilities of LMLMs, achieving the best performance on multiple tasks, including molecule captioning, computed property QA, descriptive property QA, motif counting, and IUPAC name prediction.