mlsanityData to 14 Oct 2026

Diagnosing Robotics Systems Issues with Large Language Models

arXiv 2410.09084 ↗First posted 6 Oct 2024cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

14 of 32 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 142
  10. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 52
  11. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  12. 2021LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Modelcited by 30
2 more
  1. 2021Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Modelscited by 12
  2. 2023Learning Representations on Logs for AIOpscited by 6

Built on by

Not cited yet.