mlsanityData to 14 Oct 2026

FEABench: Evaluating Language Models on Multiphysics Reasoning Ability

arXiv 2504.06260 ↗First posted 8 Apr 2025cs.AI cs.CL cs.NAcited by 22

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Building precise simulations of the real world and invoking numerical solvers to answer quantitative problems is an essential requirement in engineering and science. We present FEABench, a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). We introduce a comprehensive evaluation scheme to investigate the ability of LLMs to solve these problems end-to-end by reasoning over natural language problem descriptions and operating COMSOL Multiphysics^circledR, an FEA software, to compute the answers. We additionally design a language model agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solutions over multiple iterations. Our best performing strategy generates executable API calls 88% of the time. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would push the frontiers of automation in engineering. Acquiring this capability would augment LLMs' reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world. The code is available at https://github.com/google/feabench

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 55 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

25 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics19 Feb

Other references · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  12. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
12 more
  1. 2024Capabilities of Gemini Models in Medicinecited by 237
  2. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  3. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  4. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  5. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  6. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  7. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 48
  8. 2024LeanAgent: Lifelong Learning for Formal Theorem Provingcited by 35
  9. 2024Quantum Many-Body Physics Calculations with Large Language Modelscited by 17
  10. 2024PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Modelscited by 15
  11. 2023MyCrunchGPT: A chatGPT assisted framework for scientific machine learningcited by 8
  12. 2024Collaboration Dynamics and Reliability Challenges of Multi-Agent LLM Systems in Finite Element Analysiscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Foam-Agent 2.0: An End-to-End Composable Multi-Agent Framework for Automating CFD Simulation in OpenFOAMcited by 15
  2. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 12
  3. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problemscited by 7
  4. 2025SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoningcited by 7
  5. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoningcited by 5
  6. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  7. 2025AI Benchmark Democratization and Carpentrycited by 3
  8. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  9. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  10. 2025An MLCommons Scientific Benchmarks Ontologycited by 2
  11. 2026SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Sciencecited by 2
  12. 2026Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewardscited by 2
10 more
  1. 2026SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoningcited by 1
  2. 2026CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automationcited by 0
  3. 2026VFEAgent: A Multimodal Agent Framework for End-to-End Automated Finite Element Analysiscited by 0
  4. 2026A Constrained Natural-Language Interface for Variational Multi-Physics Finite Element Simulations in FEniCScited by 0
  5. 2026Large language models for partial differential equation workflowscited by 0
  6. 2026Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modelingcited by 0
  7. 2026What Do CAE Simulation Agents Really Need Beyond a Generic Harness?cited by 0
  8. 2026EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?cited by 0
  9. 2026Language models can notice an impossible engineering problem yet still report it as solvedcited by 0
  10. 2026EvoSim: Learning to Model, Modeling to Learncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
  2. 2026ALL-FEM: Agentic Large Language models Fine-tuned for Finite Element Methods
  3. 2025FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
  4. 2025MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulation
  5. 2025FeaGPT: an End-to-End agentic-AI for Finite Element Analysis
  6. 2026PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems
  7. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
  8. 2026Infinite Problem Generator: Verifiably Scaling Physics Reasoning Data with Agentic Workflows