# Combining TSL and LLM to Automate REST API Testing: A Comparative Study

arXiv: https://arxiv.org/abs/2509.05540 · HTML: https://mlsanity.com/paper/2509.05540/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 50 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 9 of 9)

- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [A Closer Look at System Prompt Robustness](https://mlsanity.com/paper/2502.12197/) (2025)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [SPRIG: Improving Large Language Model Performance by System Prompt Optimization](https://mlsanity.com/paper/2410.14826/) (2024)
- [Leveraging Large Language Models to Improve REST API Testing](https://mlsanity.com/paper/2312.00894/) (2023)
- [Testing RESTful APIs: A Survey](https://mlsanity.com/paper/2212.14604/) (2022)

## Built on by (most cited first, 3 of 3)

- [Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding](https://mlsanity.com/paper/2605.09271/) (2026)
- [MASTEST: A LLM-Based Multi-Agent System For RESTful API Tests](https://mlsanity.com/paper/2511.18038/) (2025)
- [RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements](https://mlsanity.com/paper/2604.25862/) (2026)

## Abstract (arXiv)

The effective execution of tests for REST APIs remains a considerable challenge for development teams, driven by the inherent complexity of distributed systems, the multitude of possible scenarios, and the limited time available for test design. Exhaustive testing of all input combinations is impractical, often resulting in undetected failures, high manual effort, and limited test coverage. To address these issues, we introduce RestTSLLM, an approach that uses Test Specification Language (TSL) in conjunction with Large Language Models (LLMs) to automate the generation of test cases for REST APIs. The approach targets two core challenges: the creation of test scenarios and the definition of appropriate input data. The proposed solution integrates prompt engineering techniques with an automated pipeline to evaluate various LLMs on their ability to generate tests from OpenAPI specifications. The evaluation focused on metrics such as success rate, test coverage, and mutation score, enabling a systematic comparison of model performance. The results indicate that the best-performing LLMs - Claude 3.5 Sonnet (Anthropic), Deepseek R1 (Deepseek), Qwen 2.5 32b (Alibaba), and Sabia 3 (Maritaca) - consistently produced robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric, emerging in this study as the most suitable model for this task. These findings highlight the potential of LLMs to automate the generation of tests based on API specifications.