# HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science

arXiv: https://arxiv.org/abs/2310.08511 · HTML: https://mlsanity.com/paper/2310.08511/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling](https://mlsanity.com/paper/2305.08264/) (2023)
- [DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articles](https://mlsanity.com/paper/2207.01079/) (2022)

## Built on by (most cited first, 22 of 22)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents](https://mlsanity.com/paper/2501.13299/) (2025)
- [SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation](https://mlsanity.com/paper/2405.09939/) (2024)
- [MatExpert: Decomposing Materials Discovery by Mimicking Human Experts](https://mlsanity.com/paper/2410.21317/) (2024)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [MatViX: Multimodal Information Extraction from Visually Rich Articles](https://mlsanity.com/paper/2410.20494/) (2024)
- [MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Design](https://mlsanity.com/paper/2412.16270/) (2024)
- [A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools](https://mlsanity.com/paper/2506.20743/) (2025)
- [MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science](https://mlsanity.com/paper/2410.15126/) (2024)
- [Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health](https://mlsanity.com/paper/2412.12981/) (2024)
- [ScienceMeter: Tracking Scientific Knowledge Updates in Language Models](https://mlsanity.com/paper/2505.24302/) (2025)
- [MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge](https://mlsanity.com/paper/2505.23982/) (2025)
- [Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite Solar Cell Research](https://mlsanity.com/paper/2502.12669/) (2025)
- [Incorporating Domain Knowledge into Materials Tokenization](https://mlsanity.com/paper/2506.11115/) (2025)
- [Quokka: An Open-source Large Language Model ChatBot for Material Science](https://mlsanity.com/paper/2401.01089/) (2024)
- [Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery](https://mlsanity.com/paper/2608.06931/) (2026)
- [Diverse LLMs or Diverse Question Interpretations? That is the Ensembling Question](https://mlsanity.com/paper/2507.21168/) (2025)
- [Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis](https://mlsanity.com/paper/2509.25281/) (2025)
- [Exploring LLMs and RAG for Plausible and Explainable Material Prediction of Vehicle Components](https://mlsanity.com/paper/2609.18437/) (2026)

## Abstract (arXiv)

We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available at urlhttps://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee.