# Multimodal machine learning for materials science: composition-structure bimodal learning for experimentally measured properties

arXiv: https://arxiv.org/abs/2309.04478 · HTML: https://mlsanity.com/paper/2309.04478/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 36 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 4 of 4)

- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)

## Built on by (most cited first, 4 of 4)

- [UniMat: Unifying Materials Embeddings through Multi-modal Learning](https://mlsanity.com/paper/2411.08664/) (2024)
- [Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning](https://mlsanity.com/paper/2506.13051/) (2025)
- [MEIDNet: Multimodal generative AI framework for inverse materials design](https://mlsanity.com/paper/2601.22009/) (2026)
- [Developing a Multi-Modal Machine Learning Model For Predicting Performance of Automotive Hood Frames](https://mlsanity.com/paper/2508.20358/) (2025)

## Abstract (arXiv)

The widespread application of multimodal machine learning models like GPT-4 has revolutionized various research fields including computer vision and natural language processing. However, its implementation in materials informatics remains underexplored, despite the presence of materials data across diverse modalities, such as composition and structure. The effectiveness of machine learning models trained on large calculated datasets depends on the accuracy of calculations, while experimental datasets often have limited data availability and incomplete information. This paper introduces a novel approach to multimodal machine learning in materials science via composition-structure bimodal learning. The proposed COmposition-Structure Bimodal Network (COSNet) is designed to enhance learning and predictions of experimentally measured materials properties that have incomplete structure information. Bimodal learning significantly reduces prediction errors across distinct materials properties including Li conductivity in solid electrolyte, band gap, refractive index, dielectric constant, energy, and magnetic moment, surpassing composition-only learning methods. Furthermore, we identified that data augmentation based on modal availability plays a pivotal role in the success of bimodal learning.