ScriptSmith: A Unified LLM Framework for Enhancing IT Operations via Automated Bash Script Generation, Assessment, and Refinement
Not on the map. Why?
Abstract (arXiv)
In the rapidly evolving landscape of site reliability engineering (SRE), the demand for efficient and effective solutions to manage and resolve issues in site and cloud applications is paramount. This paper presents an innovative approach to action automation using large language models (LLMs) for script generation, assessment, and refinement. By leveraging the capabilities of LLMs, we aim to significantly reduce the human effort involved in writing and debugging scripts, thereby enhancing the productivity of SRE teams. Our experiments focus on Bash scripts, a commonly used tool in SRE, and involve the CodeSift dataset of 100 tasks and the InterCode dataset of 153 tasks. The results show that LLMs can automatically assess and refine scripts efficiently, reducing the need for script validation in an execution environment. Results demonstrate that the framework shows an overall improvement of 7-10% in script generation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 20 references that are in the map. First posted on arXiv 12 Sep 2024.
Builds on
16 of 20 references.
References · 16
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2023Teaching Large Language Models to Self-Debugcited by 963
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
- 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
- 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
4 more
- 2019SPoC: Search-based Pseudocode to Codecited by 176
- 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
- 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
- 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
Built on by
Citing papers per year: 2025: 2.
2 papers.