# Evaluating Large Language Models for Real-World Engineering Tasks

arXiv: https://arxiv.org/abs/2505.13484 · HTML: https://mlsanity.com/paper/2505.13484/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [LLM4DyG: Can Large Language Models Solve Spatial-Temporal Problems on Dynamic Graphs?](https://mlsanity.com/paper/2310.17110/) (2023)
- [Large Language Models in Fault Localisation](https://mlsanity.com/paper/2308.15276/) (2023)
- [Toward automatic generation of control structures for process flow diagrams with large language models](https://mlsanity.com/paper/2211.05583/) (2022)
- [Control Industrial Automation System with Large Language Model Agents](https://mlsanity.com/paper/2409.18009/) (2024)
- [A Research Agenda for AI Planning in the Field of Flexible Production Systems](https://mlsanity.com/paper/2112.15484/) (2021)
- [Exploring LLMs for Verifying Technical System Specifications Against Requirements](https://mlsanity.com/paper/2411.11582/) (2024)
- [Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems](https://mlsanity.com/paper/2405.18580/) (2024)
- [Approximate Model-Based Diagnosis Using Greedy Stochastic Search](https://mlsanity.com/paper/1401.3848/) (2014)
- [Resiliency Analysis of LLM generated models for Industrial Automation](https://mlsanity.com/paper/2308.12129/) (2023)

## Built on by (most cited first, 7 of 7)

- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [Toward Autonomous Engineering Design: A Knowledge-Guided Multi-Agent Framework](https://mlsanity.com/paper/2511.03179/) (2025)
- [CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance](https://mlsanity.com/paper/2508.20420/) (2025)
- [Can LLM Coding Agents Reason About Time Series?](https://mlsanity.com/paper/2606.16545/) (2026)
- [WeirNet: A Large-Scale 3D CFD Benchmark for Geometric Surrogate Modeling of Piano Key Weirs](https://mlsanity.com/paper/2602.20714/) (2026)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)
- [Neurosymbolics for Data Engineering: Achieving Long Context Token Reduction Without Finetuning](https://mlsanity.com/paper/2609.00367/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are transformative not only for daily activities but also for engineering tasks. However, current evaluations of LLMs in engineering exhibit two critical shortcomings: (i) the reliance on simplified use cases, often adapted from examination materials where correctness is easily verifiable, and (ii) the use of ad hoc scenarios that insufficiently capture critical engineering competencies. Consequently, the assessment of LLMs on complex, real-world engineering problems remains largely unexplored. This paper addresses this gap by introducing a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios, systematically designed to cover core competencies such as product design, prognosis, and diagnosis. Using this dataset, we evaluate four state-of-the-art LLMs, including both cloud-based and locally hosted instances, to systematically investigate their performance on complex engineering tasks. Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.