mlsanityData to 14 Oct 2026

ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

arXiv 2604.19758 ↗First posted 25 Mar 2026cs.AI cs.CL cs.LGcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We present ThermoQA, a benchmark of 293 open-ended engineering thermodynamics problems in three tiers: property lookups (110 Q), component analysis (101 Q), and full cycle analysis (82 Q). Ground truth is computed programmatically from CoolProp 7.2.0, covering water, R-134a, and variable-cp air. Six frontier LLMs are evaluated across three independent runs each. The composite leaderboard is led by Claude Opus 4.6 (94.1%), GPT-5.4 (93.1%), and Gemini 3.1 Pro (92.5%). Cross-tier degradation ranges from 2.8 pp (Opus) to 32.5 pp (MiniMax), confirming that property memorization does not imply thermodynamic reasoning. Supercritical water, R-134a refrigerant, and combined-cycle gas turbine analysis serve as natural discriminators with 40-60 pp performance spreads. Multi-run sigma ranges from +/-0.1% to +/-2.5%, quantifying reasoning consistency as a distinct evaluation axis. Dataset and code are open-source at https://huggingface.co/datasets/olivenet/thermoqa

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 8 of its 18 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

8 of 18 references.

References · 8

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  4. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  5. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 12
  6. 2025Using Large Language Models for Solving Thermodynamic Problemscited by 4
  7. 2025From Canonical to Complex: Benchmarking LLM Capabilities in Undergraduate Thermodynamicscited by 2
  8. 2025Superstudent intelligence in thermodynamicscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Reasoning Reliability in Artificial Intelligence Models for Energy-System Analysis
  2. 2026Stan: An LLM-based thermodynamics course assistant
  3. 2025Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research
  4. 2024Domain-specific ReAct for physics-integrated iterative modeling: A case study of LLM agents for gas path analysis of gas turbines
  5. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning
  6. 2026Where Larger Models Excel: The Primacy of Constraint-Guided Reasoning
  7. 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation
  8. 2025Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models