Transformers Discover Molecular Structure Without Graph Priors

Computational simulations play a central role in scientific discovery, and machine learning (ML) has emerged as a promising alternative to traditional physics-based modeling. However, scientific modeling requires physically meaningful predictions, raising a fundamental question for data-driven methods: to what extent can physical inductive biases-that is, prior assumptions about the structure of the physical world-emerge by learning from data alone? In this work, we study atomistic modeling, a representative field in the computational sciences where ML architectures have historically embedded strong physical inductive biases-such as geometric locality and graph structure. We develop a systematic understanding of how physical patterns can alternatively be discovered directly from data by training a model without domain-specific priors, including any manually defined atomistic pairwise interactions. We find that the model autonomously recovers key physical structure, such as learned interatomic interaction strengths that mirror classical electrostatics and interaction cutoffs consistent with traditional physical models. We further demonstrate predictable neural scaling law behavior with increased data and compute, and find accuracy on certain metrics competitive with physics-informed architectures. Our results clarify the boundary between engineered inductive biases and learnable physical structure, suggesting that general-purpose architectures can serve as principled baselines for scientific modeling by learning fundamental physical structure directly from data.