mlsanityData to 14 Oct 2026

FactoryBench: Evaluating Industrial Machine Understanding

arXiv 2605.07675 ↗First posted 8 May 2026cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs are organized along four causal levels (state, intervention, counterfactual, decision) instantiating Pearl's ladder of causation, and span five answer formats: four structured formats are scored deterministically and free-form answers are scored by an LLM-as-judge voting protocol. We propose a scalable Q&A generation framework built around structured question templates, present FactoryWave (a dense, multitask, multivariate sensor dataset collected from a UR3 cobot and a KUKA KR10 industrial arm), and construct FactoryBench as a large-scale benchmark of over 70k Q&A items grounded in roughly 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD. Zero-shot evaluation of six frontier LLMs shows that no model exceeds 50% on structured levels or 18% on decision-making, revealing a wide gap between current models and operational machine understanding.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  12. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
21 more
  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  3. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 761
  4. 2024Chronos: Learning the Language of Time Seriescited by 759
  5. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  6. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  7. 2023A decoder-only foundation model for time-series forecastingcited by 692
  8. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  9. 2022Transformers in Time Series: A Surveycited by 497
  10. 2018The UCR Time Series Archivecited by 431
  11. 2019Deep Learning for Anomaly Detection: A Surveycited by 358
  12. 2023Tool Learning with Foundation Modelscited by 302
  13. 2017Detecting causal associations in large nonlinear time series datasetscited by 244
  14. 2020Deep Structural Causal Models for Tractable Counterfactual Inferencecited by 211
  15. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
  16. 2015Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmarkcited by 88
  17. 2025ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Datasetcited by 37
  18. 2020A Ladder of Causal Distancescited by 7
  19. 2023The voraus-AD Dataset for Anomaly Detection in Robot Applicationscited by 7
  20. 2021AURSAD: Universal Robot Screwdriving Anomaly Detection Datasetcited by 5
  21. 2025QuAnTS: Question Answering on Time Seriescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Seriescited by 2
  2. 2026Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliabilitycited by 0