LLM-Supported Natural Language to Bash Translation
Not on the map. Why?
Abstract (arXiv)
The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition circumvents these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning, and constrained decoding can improve NL2SH accuracy by up to 32%. Our findings emphasize the importance of dataset quality, execution-based evaluation and translation method for advancing NL2SH translation. Our code is available at https://github.com/westenfelder/NL2SH
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 48 references that are in the map. First posted on arXiv 7 Feb 2025.
Builds on
20 of 48 references.
References · 20
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
- 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
- 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
- 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
- 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 44
- 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 32
- 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
- 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 15
8 more
- 2024What can Large Language Models Capture about Code Functional Equivalence?cited by 14
- 2021NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commandscited by 7
- 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
- 2024Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codescited by 5
- 2024CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validationcited by 4
- 2023NL2CMD: An Updated Workflow for Natural Language to Bash Commands Translationcited by 3
- 2024ScriptSmith: A Unified LLM Framework for Enhancing IT Operations via Automated Bash Script Generation, Assessment, and Refinementcited by 3
- 2024Execution-Based Evaluation of Natural Language to Bash and PowerShell for Incident Remediationcited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.