# Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge

arXiv: https://arxiv.org/abs/2502.16451 · HTML: https://mlsanity.com/paper/2502.16451/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures](https://mlsanity.com/paper/1909.00949/) (2019)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Data-driven discovery of novel 2D materials by deep generative models](https://mlsanity.com/paper/2206.12159/) (2022)
- [CrysMMNet: Multimodal Representation for Crystal Property Prediction](https://mlsanity.com/paper/2307.05390/) (2023)
- [Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling](https://mlsanity.com/paper/2308.02165/) (2023)
- [Looking Through Glass: Knowledge Discovery from Materials Science Literature using Natural Language Processing](https://mlsanity.com/paper/2101.01508/) (2021)
- [CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision](https://mlsanity.com/paper/2112.07133/) (2021)
- [Graph Contrastive Learning for Materials](https://mlsanity.com/paper/2211.13408/) (2022)
- [Prediction of the electron density of states for crystalline compounds with Atomistic Line Graph Neural Networks (ALIGNN)](https://mlsanity.com/paper/2201.08348/) (2022)
- [1.5 million materials narratives generated by chatbots](https://mlsanity.com/paper/2308.13687/) (2023)

## Built on by (most cited first, 1 of 1)

- [MatBind: A Shared Embedding Space for Multimodal Materials Characterization](https://mlsanity.com/paper/2607.08470/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) is increasingly used for the inverse design of materials, such as crystals and molecules. Existing AI research on molecules has integrated chemical structures of molecules with textual knowledge to adapt to complex instructions. However, this approach has been unattainable for crystals due to data scarcity from the biased distribution of investigated crystals and the lack of semantic supervision in peer-reviewed literature. In this work, we introduce a contrastive language-crystals model (CLaC) pre-trained on a newly synthesized dataset of 126k crystal structure-text pairs. To demonstrate the advantage of using synthetic data to overcome data scarcity, we constructed a comparable dataset extracted from academic papers. We evaluate CLaC's generalization ability through various zero-shot cross-modal tasks and downstream applications. In experiments, CLaC achieves state-of-the-art zero-shot generalization performance in understanding crystal structures, surpassing latest large language models.