# MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalities

arXiv: https://arxiv.org/abs/2310.10445 · HTML: https://mlsanity.com/paper/2310.10445/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems](https://mlsanity.com/paper/2306.17525/) (2023)
- [Generative Pretrained Autoregressive Transformer Graph Neural Network applied to the Analysis and Discovery of Novel Proteins](https://mlsanity.com/paper/2305.04934/) (2023)
- [Borges and AI](https://mlsanity.com/paper/2310.01425/) (2023)

## Built on by (most cited first, 30 of 38)

- [Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap](https://mlsanity.com/paper/2401.10034/) (2024)
- [Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities](https://mlsanity.com/paper/2409.03444/) (2024)
- [ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning](https://mlsanity.com/paper/2402.04268/) (2024)
- [MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge](https://mlsanity.com/paper/2311.08166/) (2023)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design](https://mlsanity.com/paper/2402.07148/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence](https://mlsanity.com/paper/2407.10022/) (2024)
- [Large Language Model Agent as a Mechanical Designer](https://mlsanity.com/paper/2404.17525/) (2024)
- [LLM-3D Print: Large Language Models To Monitor and Control 3D Printing](https://mlsanity.com/paper/2408.14307/) (2024)
- [PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking](https://mlsanity.com/paper/2410.12375/) (2024)
- [Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding](https://mlsanity.com/paper/2401.12983/) (2024)
- [Foam-Agent 2.0: An End-to-End Composable Multi-Agent Framework for Automating CFD Simulation in OpenFOAM](https://mlsanity.com/paper/2509.18178/) (2025)
- [Autonomous Agents Coordinating Distributed Discovery Through Emergent Artifact Exchange](https://mlsanity.com/paper/2603.14312/) (2026)
- [Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design](https://mlsanity.com/paper/2405.19076/) (2024)
- [Foam-Agent: A Large Language Model-Based Multi-Agent Framework for Automating Computational Fluid Dynamics Workflows](https://mlsanity.com/paper/2505.04997/) (2025)
- [ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models](https://mlsanity.com/paper/2505.12534/) (2025)
- [GraphAgents: Knowledge Graph-Guided Agentic AI for Cross-Domain Materials Design](https://mlsanity.com/paper/2602.07491/) (2026)
- [Retrieval Augmented Thought Process for Private Data Handling in Healthcare](https://mlsanity.com/paper/2402.07812/) (2024)
- [Learning the rules of peptide self-assembly through data mining with large language models](https://mlsanity.com/paper/2411.05421/) (2024)
- [Large Language Models for Extrapolative Modeling of Manufacturing Processes](https://mlsanity.com/paper/2502.12185/) (2025)
- [From Perception to Autonomous Computational Modeling: A Multi-Agent Approach](https://mlsanity.com/paper/2604.06788/) (2026)
- [Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence](https://mlsanity.com/paper/2606.01444/) (2026)
- [Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination](https://mlsanity.com/paper/2607.00924/) (2026)
- [Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models](https://mlsanity.com/paper/2502.14904/) (2025)
- [AutomataGPT: Forecasting and Ruleset Inference for Two-Dimensional Cellular Automata](https://mlsanity.com/paper/2506.17333/) (2025)
- [Evaluating Large Language Models for Material Selection](https://mlsanity.com/paper/2405.03695/) (2024)
- [AutoMOOSE: An Agentic AI for Autonomous Phase-Field Simulation](https://mlsanity.com/paper/2603.20986/) (2026)
- [MusicSwarm: Biologically Inspired Intelligence for Music Composition](https://mlsanity.com/paper/2509.11973/) (2025)
- [From Papers to Property Tables: A Priority-Based LLM Workflow for Materials Data Extraction](https://mlsanity.com/paper/2604.07584/) (2026)

## Abstract (arXiv)

For centuries, researchers have sought out ways to connect disparate areas of knowledge. While early scholars (Galileo, da Vinci, etc.) were experts across fields, specialization has taken hold later. With the advent of Artificial Intelligence, we can now explore relationships across areas (e.g., mechanics-biology) or disparate domains (e.g., failure mechanics-art). To achieve this, we use a fine-tuned Large Language Model (LLM), here for a subset of knowledge in multiscale materials failure. The approach includes the use of a general-purpose LLM to distill question-answer pairs from raw sources followed by LLM fine-tuning. The resulting MechGPT LLM foundation model is used in a series of computational experiments to explore its capacity for knowledge retrieval, various language tasks, hypothesis generation, and connecting knowledge across disparate areas. While the model has some ability to recall knowledge from training, we find that LLMs are particularly useful to extract structural insights through Ontological Knowledge Graphs. These interpretable graph structures provide explanatory insights, frameworks for new research questions, and visual representations of knowledge that also can be used in retrieval-augmented generation. Three versions of MechGPT are discussed, featuring different sizes from 13 billion to 70 billion parameters, and reaching context lengths of more than 10,000 tokens. This provides ample capacity for sophisticated retrieval augmented strategies, as well as agent-based modeling where multiple LLMs interact collaboratively and/or adversarially, the incorporation of new data from the literature or web searches, as well as multimodality.