# FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

arXiv: https://arxiv.org/abs/2512.20732 · HTML: https://mlsanity.com/paper/2512.20732/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 79 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 18 of 18)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models](https://mlsanity.com/paper/2404.05221/) (2024)
- [Evaluating Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2512.15567/) (2025)
- [CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning](https://mlsanity.com/paper/2503.13517/) (2025)
- [AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study](https://mlsanity.com/paper/2504.08846/) (2025)

## Built on by (most cited first, 8 of 8)

- [An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc](https://mlsanity.com/paper/2603.15976/) (2026)
- [Large Language Models and their Awareness of Mechanics and Spatial Geometry](https://mlsanity.com/paper/2608.14615/) (2026)
- [VFEAgent: A Multimodal Agent Framework for End-to-End Automated Finite Element Analysis](https://mlsanity.com/paper/2605.28978/) (2026)
- [From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis](https://mlsanity.com/paper/2607.24459/) (2026)
- [Large language models for partial differential equation workflows](https://mlsanity.com/paper/2608.03600/) (2026)
- [What Do CAE Simulation Agents Really Need Beyond a Generic Harness?](https://mlsanity.com/paper/2609.03718/) (2026)
- [EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning](https://mlsanity.com/paper/2609.33261/) (2026)
- [Language models can notice an impossible engineering problem yet still report it as solved](https://mlsanity.com/paper/2610.06668/) (2026)

## Abstract (arXiv)

As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.