# Addressing the Accuracy-Cost Tradeoff in Material Property Prediction: A Teacher-Student Strategy

arXiv: https://arxiv.org/abs/2309.04482 · HTML: https://mlsanity.com/paper/2309.04482/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [Adapting Models to Signal Degradation using Distillation](https://mlsanity.com/paper/1604.00433/) (2016)
- [Connectivity Optimized Nested Graph Networks for Crystal Structures](https://mlsanity.com/paper/2302.14102/) (2023)

## Abstract (arXiv)

Deep learning has revolutionized the process of new material discovery, with state-of-the-art models now able to predict material properties based solely on chemical compositions, thus eliminating the necessity for material structures. However, this cost-effective method has led to a trade-off in model accuracy. Specifically, the accuracy of Chemical Composition-based Property Prediction Models (CPMs) significantly lags behind that of Structure-based Property Prediction Models (SPMs). To tackle this challenge, we propose an innovative Teacher-Student (T-S) strategy, where a pre-trained SPM serves as the 'teacher' to enhance the accuracy of the CPM. Leveraging the T-S strategy, T-S CrabNet has risen to become the most accurate model among current CPMs. Initially, we demonstrated the universality of this strategy. On the Materials Project (MP) and Jarvis datasets, we validated the effectiveness of the T-S strategy in boosting the accuracy of CPMs with two distinct network structures, namely CrabNet and Roost. This led to CrabNet, under the guidance of the T-S strategy, emerging as the most accurate model among the current CPMs. Moreover, this strategy shows remarkable efficacy in small datasets. When predicting the formation energy on a small MP dataset comprising merely 5% of the samples, the T-S strategy boosted CrabNet's accuracy by 37.1%, exceeding the enhancement effect of the T-S strategy on the whole dataset.