# Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

arXiv: https://arxiv.org/abs/2511.12036 · HTML: https://mlsanity.com/paper/2511.12036/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Preference optimization for molecular design, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 16 of its 61 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference optimization for molecular design](https://mlsanity.com/alignment/preference-optimization-for-molecular-design/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms](https://mlsanity.com/paper/2406.02900/) (2024)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions](https://mlsanity.com/paper/2410.23405/) (2024)
- [LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization](https://mlsanity.com/paper/2505.14756/) (2025)
- [PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Design](https://mlsanity.com/paper/2509.07150/) (2025)
- [Generative Hierarchical Materials Search](https://mlsanity.com/paper/2409.06762/) (2024)
- [Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation](https://mlsanity.com/paper/2503.00152/) (2025)
- [Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2505.16379/) (2025)
- [PriM: Principle-Inspired Material Discovery through Multi-Agent Collaboration](https://mlsanity.com/paper/2504.08810/) (2025)

## Built on by (most cited first, 1 of 1)

- [Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery](https://mlsanity.com/paper/2605.08956/) (2026)

## Abstract (arXiv)

We apply preference learning to the task of language model-guided design of novel structural alloys. In contrast to prior work that focuses on generating stable inorganic crystals, our approach targets the synthesizeability of a specific structural class: BCC/B2 superalloys, an underexplored family of materials with potential applications in extreme environments. Using three open-weight models (LLaMA-3.1, Gemma-2, and OLMo-2), we demonstrate that language models can be optimized for multiple design objectives using a single, unified reward signal through Direct Preference Optimization (DPO). Unlike prior approaches that rely on heuristic or human-in-the-loop feedback (costly), our reward signal is derived from thermodynamic phase calculations, offering a scientifically grounded criterion for model tuning. To our knowledge, this is the first demonstration of preference-tuning a language model using physics-grounded feedback for structural alloy design. The resulting framework is general and extensible, providing a path forward for intelligent design-space exploration across a range of physical science domains.