Benchmarking Reasoning Reliability in Artificial Intelligence Models for Energy-System Analysis
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Artificial intelligence and machine learning are increasingly used for forecasting, optimization, and policy design in the energy sector, yet no standardized framework exists to evaluate whether these systems reason correctly. Current validation practices focus on predictive accuracy or computational efficiency, leaving the logical integrity of analytical conclusions untested. This study introduces the Analytical Reliability Benchmark (ARB), a reproducible framework that quantifies reasoning reliability in large language models applied to energy system analysis. The benchmark integrates five submetrics: accuracy, reasoning reliability, uncertainty discipline, policy consistency, and transparency, and evaluates model performance across deterministic, probabilistic, and epistemic scenarios using open technoeconomic datasets (NREL ATB 2024, DOE H2A/H2New, IEA WEO 2024). Four frontier models (GPT-4/5, Claude 4.5 Sonnet, Gemini 2.5 Pro, Llama 3 70B) were tested under identical factual and regulatory conditions. Results show that reasoning reliability can be objectively measured. GPT-4/5 and Claude 4.5 Sonnet achieved consistent and policy-compliant reasoning (Analytical Reliability Index greater than 90), Gemini 2.5 Pro demonstrated moderate stability, and Llama 3 70B remained below professional thresholds. Statistical validation confirmed that these differences are significant and reproducible. The ARB establishes the first quantitative method in the energy literature for verifying causal, probabilistic, and policy-driven reasoning in artificial intelligence systems, providing a reference framework for trustworthy and transparent analytical applications in the global energy transition.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 1 of its 23 references that are in the map. First posted on arXiv 16 Oct 2025.
Builds on
1 of 23 references.
References · 1
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models
- 2026How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains
- 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
- 2024ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models
- 2023ARB: Advanced Reasoning Benchmark for Large Language Models
- 2026TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems
- 2026RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
- 2025Exploratory Semantic Reliability Analysis of Wind Turbine Maintenance Logs using Large Language Models