Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents
On the map: LLM reasoning → Reasoning benchmarks and evaluation
also in 1 more
Abstract (arXiv)
The Engineering Reasoning and Instruction (ERI) benchmark is a taxonomy-driven instruction dataset designed to train and evaluate engineering-capable large language models (LLMs) and agents. This dataset spans nine engineering fields (namely: civil, mechanical, electrical, chemical, environmental, aerospace, materials, fire, and industrial engineering) and 55 subdomains, and is crossed with seven intent types (i.e., definition, explanation, calculation, comparison, design/synthesis, troubleshooting, and code-related) and three difficulty tiers (undergraduate, graduate, and professional), yielding 57,750 records with field/subdomain/type/difficulty metadata and solution formatting. We examined ERI via seven LLMs and report a statistically significant three-tier performance structure, with frontier models (GPT-5, Claude Sonnet 4, DeepSeek V3.1) achieving mean scores above 4.30 on a five-point scale, while mid-tier and smaller models exhibited progressively higher failure rates and steeper performance degradation on graduate-level questions. To address circularity concerns inherent in LLM benchmarks, we developed a convergent validation protocol that leverages cross-provider independence, multi-judge averaging, and frontier-model agreement analysis to empirically bound hallucination risk to 1.7%. ERI is released with taxonomy specifications, validation scripts, and an evaluation harness to enable reproducible comparisons and regression testing for instruction tuning, routing, retrieval-augmented evaluation, and agentic tool-use workflows in engineering settings.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 29 of its 42 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
29 of 42 references.
Closest ancestors · 1
Same direction, newest first
Other references · 28
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
16 more
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
- 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
- 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
- 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
- 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
- 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
- 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
- 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
- 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48
- 2025Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Taskscited by 17
- 2025Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMscited by 14
- 2026AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understandingcited by 8
- 2025SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materialscited by 7
- 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6
- 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
- 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
- 2024MAmmoTH2: Scaling Instructions from the Web
- 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
- 2026Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain
- 2026Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation
- 2026Logics-STEM: Empowering LLM Reasoning via Failure-Driven Post-Training and Document Knowledge Enhancement
- 2025Big Reasoning with Small Models: Instruction Retrieval at Inference Time