# Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning

arXiv: https://arxiv.org/abs/2506.13051 · HTML: https://mlsanity.com/paper/2506.13051/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [FAENet: Frame Averaging Equivariant GNN for Materials Modeling](https://mlsanity.com/paper/2305.05577/) (2023)
- [ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area](https://mlsanity.com/paper/2408.07246/) (2024)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [DenseGNN: universal and scalable deeper graph neural networks for high-performance property prediction in crystals and molecules](https://mlsanity.com/paper/2501.03278/) (2025)
- [xChemAgents: Agentic AI for Explainable Quantum Chemistry](https://mlsanity.com/paper/2505.20574/) (2025)
- [Multimodal machine learning for materials science: composition-structure bimodal learning for experimentally measured properties](https://mlsanity.com/paper/2309.04478/) (2023)

## Built on by (most cited first, 4 of 4)

- [AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials](https://mlsanity.com/paper/2510.04704/) (2025)
- [SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models](https://mlsanity.com/paper/2601.22312/) (2026)
- [ChronoVision: Temporal Reasoning via Latent State Reconstruction](https://mlsanity.com/paper/2608.05631/) (2026)
- [Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures](https://mlsanity.com/paper/2609.00663/) (2026)

## Abstract (arXiv)

Evaluating foundation models for crystallographic reasoning requires benchmarks that isolate generalization behavior while enforcing physical constraints. This work introduces a multiscale multicrystal dataset with two physically grounded evaluation protocols to stress-test multimodal generative models. The Spatial-Exclusion benchmark withholds all supercells of a given radius from a diverse dataset, enabling controlled assessments of spatial interpolation and extrapolation. The Compositional-Exclusion benchmark omits all samples of a specific chemical composition, probing generalization across stoichiometries. Nine vision--language foundation models are prompted with crystallographic images and textual context to generate structural annotations. Responses are evaluated via (i) relative errors in lattice parameters and density, (ii) a physics-consistency index penalizing volumetric violations, and (iii) a hallucination score capturing geometric outliers and invalid space-group predictions. These benchmarks establish a reproducible, physically informed framework for assessing generalization, consistency, and reliability in large-scale multimodal models. Dataset and code are available at https://github.com/KurbanIntelligenceLab/StressTestingMMFMinCR.