# MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems

arXiv: https://arxiv.org/abs/2306.17525 · HTML: https://mlsanity.com/paper/2306.17525/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 86 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [Machine learning for molecular simulation](https://mlsanity.com/paper/1911.02792/) (2019)
- [A very preliminary analysis of DALL-E 2](https://mlsanity.com/paper/2204.13807/) (2022)
- [Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model](https://mlsanity.com/paper/1609.00680/) (2016)
- [Deep autoencoders for physics-constrained data-driven nonlinear materials modeling](https://mlsanity.com/paper/2209.04416/) (2022)
- [Generative Pretrained Autoregressive Transformer Graph Neural Network applied to the Analysis and Discovery of Novel Proteins](https://mlsanity.com/paper/2305.04934/) (2023)

## Built on by (most cited first, 29 of 29)

- [Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities](https://mlsanity.com/paper/2409.03444/) (2024)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning](https://mlsanity.com/paper/2402.04268/) (2024)
- [MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge](https://mlsanity.com/paper/2311.08166/) (2023)
- [From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design](https://mlsanity.com/paper/2311.12668/) (2023)
- [MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalities](https://mlsanity.com/paper/2310.10445/) (2023)
- [AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence](https://mlsanity.com/paper/2407.10022/) (2024)
- [BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials](https://mlsanity.com/paper/2309.08788/) (2023)
- [PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking](https://mlsanity.com/paper/2410.12375/) (2024)
- [Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding](https://mlsanity.com/paper/2401.12983/) (2024)
- [Generative modeling, design and analysis of spider silk protein sequences for enhanced mechanical properties](https://mlsanity.com/paper/2309.10170/) (2023)
- [Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design](https://mlsanity.com/paper/2405.19076/) (2024)
- [Foam-Agent: A Large Language Model-Based Multi-Agent Framework for Automating Computational Fluid Dynamics Workflows](https://mlsanity.com/paper/2505.04997/) (2025)
- [SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials](https://mlsanity.com/paper/2509.21079/) (2025)
- [Learning Electromagnetic Metamaterial Physics With ChatGPT](https://mlsanity.com/paper/2404.15458/) (2024)
- [Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2410.13768/) (2024)
- [ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion model](https://mlsanity.com/paper/2310.10605/) (2023)
- [Micrometer: Micromechanics Transformer for Predicting Mechanical Responses of Heterogeneous Materials](https://mlsanity.com/paper/2410.05281/) (2024)
- [Automatically Differentiable Model Updating (ADiMU): conventional, hybrid, and neural network material model discovery including history-dependency](https://mlsanity.com/paper/2505.07801/) (2025)
- [Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence](https://mlsanity.com/paper/2606.01444/) (2026)
- [Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination](https://mlsanity.com/paper/2607.00924/) (2026)
- [Evaluating Large Language Models for Material Selection](https://mlsanity.com/paper/2405.03695/) (2024)
- [Autonomous Inorganic Materials Discovery via Multi-Agent Physics-Aware Scientific Reasoning](https://mlsanity.com/paper/2508.02956/) (2025)
- [Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Model](https://mlsanity.com/paper/2607.20058/) (2026)
- [Machine Learning-Guided Design of Non-Reciprocal and Asymmetric Elastic Chiral Metamaterials](https://mlsanity.com/paper/2404.13215/) (2024)
- [Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research](https://mlsanity.com/paper/2505.06964/) (2025)
- [NLP4Neuro: Sequence-to-sequence learning for neural population decoding](https://mlsanity.com/paper/2507.02264/) (2025)
- [Toward using explainable data-driven surrogate models for treating performance-based seismic design as an inverse engineering problem](https://mlsanity.com/paper/2508.00286/) (2025)
- [Self-Refining Topology Optimization via an LLM-Based Multi-Agent Framework](https://mlsanity.com/paper/2605.23273/) (2026)

## Abstract (arXiv)

We report a flexible multi-modal mechanics language model, MeLM, applied to solve various nonlinear forward and inverse problems, that can deal with a set of instructions, numbers and microstructure data. The framework is applied to various examples including bio-inspired hierarchical honeycomb design, carbon nanotube mechanics, and protein unfolding. In spite of the flexible nature of the model-which allows us to easily incorporate diverse materials, scales, and mechanical features-it performs well across disparate forward and inverse tasks. Based on an autoregressive attention-model, MeLM effectively represents a large multi-particle system consisting of hundreds of millions of neurons, where the interaction potentials are discovered through graph-forming self-attention mechanisms that are then used to identify relationships from emergent structures, while taking advantage of synergies discovered in the training data. We show that the model can solve complex degenerate mechanics design problems and determine novel material architectures across a range of hierarchical levels, providing an avenue for materials discovery and analysis. Looking beyond the demonstrations reported in this paper, we discuss other opportunities in applied mechanics and general considerations about the use of large language models in modeling, design, and analysis that can span a broad spectrum of material properties from mechanical, thermal, optical, to electronic.