mlsanityData to 7 Oct 2026

DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

arXiv 2605.08614 ↗First posted 9 May 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce ours, a benchmark of 6,690 expert-validated multiple-choice questions from 118 rule-action pairs across 16 asset types. We contribute (i) a symbolic-to-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding-based distractor sampling, (ii) five variants probing distinct failure modes (Pro, Pert, Verbose, Aug, Rationale), and (iii) a benchmark of 29 LLMs and 4 embedding baselines. A human evaluation (9 practitioners, mean 45.0%) confirms ours requires specialist knowledge beyond operational experience. Three findings stand out. The frontier has closed: the top three LLMs lie within one Macro point, with Bradley-Terry Elo placing claude-opus-4-6 30 points above the next model. Yet ours,Pro exposes brittleness, with every model losing 13--60% relative accuracy under distractor expansion. ours,Aug exposes pattern-matching: under condition inversion, frontier models still select the original answer 49--63% of the time. The deployment bottleneck is not capability but calibration: frontier models handle template-style fault detection but break under structural perturbation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

27 of 49 references.

References · 27

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2025Humanity's Last Examcited by 467
  8. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  9. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  10. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  11. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  12. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
15 more
  1. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 74
  2. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  3. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 50
  4. 2025ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Datasetcited by 35
  5. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  6. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  7. 2024ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosurescited by 12
  8. 2024CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structurescited by 11
  9. 2024RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenancecited by 7
  10. 2025WFCRL: A Multi-Agent Reinforcement Learning Benchmark for Wind Farm Controlcited by 7
  11. 2024ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environmentscited by 5
  12. 2024RuAG: Learned-rule-augmented Generation for Large Language Modelscited by 4
  13. 2025PHM-Bench: A Domain-Specific Benchmarking Framework for Systematic Evaluation of Large Models in Prognostics and Health Managementcited by 3
  14. 2025A Comparative Benchmark of Large Language Models for Labelling Wind Turbine Maintenance Logscited by 3
  15. 2025LC-Opt: Benchmarking Reinforcement Learning and Agentic AI for End-to-End Liquid Cooling Optimization in Data Centerscited by 3

Built on by

Not cited yet.