# Test-driven Software Experimentation with LASSO: an LLM Prompt Benchmarking Example

arXiv: https://arxiv.org/abs/2410.08911 · HTML: https://mlsanity.com/paper/2410.08911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [HITS: High-coverage LLM-based Unit Test Generation via Method Slicing](https://mlsanity.com/paper/2408.11324/) (2024)
- [On the Evaluation of Large Language Models in Unit Test Generation](https://mlsanity.com/paper/2406.18181/) (2024)
- [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://mlsanity.com/paper/2407.07064/) (2024)
- [Test Oracle Automation in the era of LLMs](https://mlsanity.com/paper/2405.12766/) (2024)
- [The Fault in our Stars: Quality Assessment of Code Generation Benchmarks](https://mlsanity.com/paper/2404.10155/) (2024)
- [N-Version Assessment and Enhancement of Generative AI](https://mlsanity.com/paper/2409.14071/) (2024)
- [Gamifying a Software Testing Course with Continuous Integration](https://mlsanity.com/paper/2401.17740/) (2024)

## Abstract (arXiv)

Empirical software engineering faces a critical gap: the lack of standardized tools for rapid development and execution of Test-Driven Software Experiments (TDSEs) -- that is, experiments that involve the execution of software subjects and the observation and analysis of their "de facto" run-time behavior. In this paper we present a general-purpose analysis platform called LASSO that provides a minimal set of domain-specific languages and data structures to conduct TDSEs. By empowering users with an executable scripting language to design and execute TDSEs, LASSO enables efficient evaluation of run-time semantics and execution characteristics in addition to statically determined properties. We present an example TDSE that demonstrates the practical benefits of LASSO's scripting capabilities for assessing the reliability of LLMs for code generation by means of a self-contained, reusable and extensible study script. The LASSO platform and live pipeline examples are publicly available at: https://softwareobservatorium.github.io/.