# Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction

arXiv: https://arxiv.org/abs/2312.17495 · HTML: https://mlsanity.com/paper/2312.17495/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 90 references that are in the map.
- First posted on arXiv 29 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networks](https://mlsanity.com/paper/1806.07537/) (2018)
- [Learn to Combine Modalities in Multimodal Deep Learning](https://mlsanity.com/paper/1805.11730/) (2018)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [Multi-task Joint Strategies of Self-supervised Representation Learning on Biomedical Networks for Drug Discovery](https://mlsanity.com/paper/2201.04437/) (2022)
- [Artificial Intelligence for Drug Discovery: Are We There Yet?](https://mlsanity.com/paper/2307.06521/) (2023)
- [Physical Pooling Functions in Graph Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2207.13779/) (2022)
- [Modeling Industrial ADMET Data with Multitask Networks](https://mlsanity.com/paper/1606.08793/) (2016)

## Built on by (most cited first, 4 of 4)

- [ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learning](https://mlsanity.com/paper/2510.16824/) (2025)
- [Data Fusion of Deep Learned Molecular Embeddings for Property Prediction](https://mlsanity.com/paper/2504.07297/) (2025)
- [Multi-view biomedical foundation models for molecule-target and property prediction](https://mlsanity.com/paper/2410.19704/) (2024)
- [MOLAR: Learning Multimodal Molecular Representations from Noisy Labels](https://mlsanity.com/paper/2606.18390/) (2026)

## Abstract (arXiv)

Accurately predicting molecular properties is a challenging but essential task in drug discovery. Recently, many mono-modal deep learning methods have been successfully applied to molecular property prediction. However, the inherent limitation of mono-modal learning arises from relying solely on one modality of molecular representation, which restricts a comprehensive understanding of drug molecules and hampers their resilience against data noise. To overcome the limitations, we construct multimodal deep learning models to cover different molecular representations. We convert drug molecules into three molecular representations, SMILES-encoded vectors, ECFP fingerprints, and molecular graphs. To process the modal information, Transformer-Encoder, bi-directional gated recurrent units (BiGRU), and graph convolutional network (GCN) are utilized for feature learning respectively, which can enhance the model capability to acquire complementary and naturally occurring bioinformatics information. We evaluated our triple-modal model on six molecule datasets. Different from bi-modal learning models, we adopt five fusion methods to capture the specific features and leverage the contribution of each modal information better. Compared with mono-modal models, our multimodal fused deep learning (MMFDL) models outperform single models in accuracy, reliability, and resistance capability against noise. Moreover, we demonstrate its generalization ability in the prediction of binding constants for protein-ligand complex molecules in the refined set of PDBbind. The advantage of the multimodal model lies in its ability to process diverse sources of data using proper models and suitable fusion methods, which would enhance the noise resistance of the model while obtaining data diversity.