# Generative Hierarchical Materials Search

arXiv: https://arxiv.org/abs/2409.06762 · HTML: https://mlsanity.com/paper/2409.06762/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 30 of 30)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [Translating Natural Language to Planning Goals with Large-Language Models](https://mlsanity.com/paper/2302.05128/) (2023)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [ChemLLM: A Chemical Large Language Model](https://mlsanity.com/paper/2402.06852/) (2024)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [Compositional Foundation Models for Hierarchical Planning](https://mlsanity.com/paper/2309.08587/) (2023)
- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files](https://mlsanity.com/paper/2305.05708/) (2023)
- [Scalable Diffusion for Materials Generation](https://mlsanity.com/paper/2311.09235/) (2023)
- [Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures](https://mlsanity.com/paper/1909.00949/) (2019)
- [BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials](https://mlsanity.com/paper/2309.08788/) (2023)
- [Materials science in the era of large language models: a perspective](https://mlsanity.com/paper/2403.06949/) (2024)

## Built on by (most cited first, 13 of 13)

- [MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulation](https://mlsanity.com/paper/2504.08621/) (2025)
- [Space Group Equivariant Crystal Diffusion](https://mlsanity.com/paper/2505.10994/) (2025)
- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2505.16379/) (2025)
- [MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Models](https://mlsanity.com/paper/2502.20933/) (2025)
- [Cliqueformer: Model-Based Optimization with Structured Transformers](https://mlsanity.com/paper/2410.13106/) (2024)
- [CrystalReasoner: Reasoning and RL for Property-Conditioned Crystal Structure Generation](https://mlsanity.com/paper/2605.14344/) (2026)
- [Enhancing Materials Discovery with Valence Constrained Design in Generative Modeling](https://mlsanity.com/paper/2507.19799/) (2025)
- [SLayerGen: a Crystal Generative Model for all Space and Layer Groups](https://mlsanity.com/paper/2605.08262/) (2026)
- [Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys](https://mlsanity.com/paper/2511.12036/) (2025)
- [Offline Materials Optimization with CliqueFlowmer](https://mlsanity.com/paper/2603.06082/) (2026)
- [LapidaryEngine: Fully Conversational Crystal Generation](https://mlsanity.com/paper/2606.14215/) (2026)
- [Atomistic Language Models Understand and Generate Materials](https://mlsanity.com/paper/2606.21395/) (2026)

## Abstract (arXiv)

Generative models trained at scale can now produce text, video, and more recently, scientific data such as crystal structures. In applications of generative approaches to materials science, and in particular to crystal structures, the guidance from the domain expert in the form of high-level instructions can be essential for an automated system to output candidate crystals that are viable for downstream research. In this work, we formulate end-to-end language-to-structure generation as a multi-objective optimization problem, and propose Generative Hierarchical Materials Search (GenMS) for controllable generation of crystal structures. GenMS consists of (1) a language model that takes high-level natural language as input and generates intermediate textual information about a crystal (e.g., chemical formulae), and (2) a diffusion model that takes intermediate information as input and generates low-level continuous value crystal structures. GenMS additionally uses a graph neural network to predict properties (e.g., formation energy) from the generated crystal structures. During inference, GenMS leverages all three components to conduct a forward tree search over the space of possible structures. Experiments show that GenMS outperforms other alternatives of directly using language models to generate structures both in satisfying user request and in generating low-energy structures. We confirm that GenMS is able to generate common crystal structures such as double perovskites, or spinels, solely from natural language input, and hence can form the foundation for more complex structure generation in near future.