mlsanityData to 14 Oct 2026

MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems

arXiv 2306.17525 ↗First posted 30 Jun 2023cond-mat.mtrl-sci cond-mat.dis-nn cond-mat.mes-hallcited by 29

Not on the map. Why?

Abstract (arXiv)

We report a flexible multi-modal mechanics language model, MeLM, applied to solve various nonlinear forward and inverse problems, that can deal with a set of instructions, numbers and microstructure data. The framework is applied to various examples including bio-inspired hierarchical honeycomb design, carbon nanotube mechanics, and protein unfolding. In spite of the flexible nature of the model-which allows us to easily incorporate diverse materials, scales, and mechanical features-it performs well across disparate forward and inverse tasks. Based on an autoregressive attention-model, MeLM effectively represents a large multi-particle system consisting of hundreds of millions of neurons, where the interaction potentials are discovered through graph-forming self-attention mechanisms that are then used to identify relationships from emergent structures, while taking advantage of synergies discovered in the training data. We show that the model can solve complex degenerate mechanics design problems and determine novel material architectures across a range of hierarchical levels, providing an avenue for materials discovery and analysis. Looking beyond the demonstrations reported in this paper, we discuss other opportunities in applied mechanics and general considerations about the use of large language models in modeling, design, and analysis that can span a broad spectrum of material properties from mechanical, thermal, optical, to electronic.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 30 of its 86 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

30 of 86 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2017Graph Attention Networkscited by 8,923
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
18 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020Reformer: The Efficient Transformercited by 1,668
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2022Galactica: A Large Language Model for Sciencecited by 744
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  11. 2022Transformers are Sample-Efficient World Modelscited by 328
  12. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 229
  13. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  14. 2019Machine learning for molecular simulationcited by 110
  15. 2022A very preliminary analysis of DALL-E 2cited by 82
  16. 2016Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Modelcited by 31
  17. 2022Deep autoencoders for physics-constrained data-driven nonlinear materials modelingcited by 13
  18. 2023Generative Pretrained Autoregressive Transformer Graph Neural Network applied to the Analysis and Discovery of Novel Proteinscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 12; 2025: 7; 2026: 4.

29 papers.

Sort:

By area:

  1. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
  2. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  3. 2024ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learningcited by 64
  4. 2023MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledgecited by 63
  5. 2023From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Designcited by 51
  6. 2023MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalitiescited by 38
  7. 2024AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligencecited by 36
  8. 2023BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materialscited by 30
  9. 2024PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinkingcited by 17
  10. 2024Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understandingcited by 15
  11. 2023Generative modeling, design and analysis of spider silk protein sequences for enhanced mechanical propertiescited by 9
  12. 2024Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Designcited by 9
17 more
  1. 2025Foam-Agent: A Large Language Model-Based Multi-Agent Framework for Automating Computational Fluid Dynamics Workflowscited by 9
  2. 2025SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materialscited by 8
  3. 2024Learning Electromagnetic Metamaterial Physics With ChatGPTcited by 6
  4. 2024Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systemscited by 6
  5. 2023ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion modelcited by 5
  6. 2024Micrometer: Micromechanics Transformer for Predicting Mechanical Responses of Heterogeneous Materialscited by 5
  7. 2025Automatically Differentiable Model Updating (ADiMU): conventional, hybrid, and neural network material model discovery including history-dependencycited by 5
  8. 2026Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligencecited by 4
  9. 2026Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombinationcited by 4
  10. 2024Evaluating Large Language Models for Material Selectioncited by 2
  11. 2025Autonomous Inorganic Materials Discovery via Multi-Agent Physics-Aware Scientific Reasoningcited by 1
  12. 2026Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Modelcited by 1
  13. 2024Machine Learning-Guided Design of Non-Reciprocal and Asymmetric Elastic Chiral Metamaterialscited by 0
  14. 2025Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Researchcited by 0
  15. 2025NLP4Neuro: Sequence-to-sequence learning for neural population decodingcited by 0
  16. 2025Toward using explainable data-driven surrogate models for treating performance-based seismic design as an inverse engineering problemcited by 0
  17. 2026Self-Refining Topology Optimization via an LLM-Based Multi-Agent Frameworkcited by 0