# LlamaRestTest: Effective REST API Testing with Small Language Models

arXiv: https://arxiv.org/abs/2501.08598 · HTML: https://mlsanity.com/paper/2501.08598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://mlsanity.com/paper/2306.06624/) (2023)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [HAWQV3: Dyadic Neural Network Quantization](https://mlsanity.com/paper/2011.10680/) (2020)
- [NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls](https://mlsanity.com/paper/2409.03797/) (2024)
- [Norm Tweaking: High-performance Low-bit Quantization of Large Language Models](https://mlsanity.com/paper/2309.02784/) (2023)
- [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023)
- [Leveraging Large Language Models to Improve REST API Testing](https://mlsanity.com/paper/2312.00894/) (2023)
- [How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes](https://mlsanity.com/paper/2409.03454/) (2024)
- [Automated Test Generation for REST APIs: No Time to Rest Yet](https://mlsanity.com/paper/2204.08348/) (2022)
- [Morest: Model-based RESTful API Testing with Execution Feedback](https://mlsanity.com/paper/2204.12148/) (2022)
- [Testing RESTful APIs: A Survey](https://mlsanity.com/paper/2212.14604/) (2022)
- [A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs](https://mlsanity.com/paper/2411.07098/) (2024)
- [Adaptive REST API Testing with Reinforcement Learning](https://mlsanity.com/paper/2309.04583/) (2023)
- [Open Problems in Fuzzing RESTful APIs: A Comparison of Tools](https://mlsanity.com/paper/2205.05325/) (2022)
- [QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs](https://mlsanity.com/paper/1912.09686/) (2019)
- [Language Models are Better Bug Detector Through Code-Pair Classification](https://mlsanity.com/paper/2311.07957/) (2023)
- [Automatic Property-based Testing of GraphQL APIs](https://mlsanity.com/paper/2012.07380/) (2020)

## Built on by (most cited first, 4 of 4)

- [Evaluating LLMs on Sequential API Call Through Automated Test Generation](https://mlsanity.com/paper/2507.09481/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap](https://mlsanity.com/paper/2505.19625/) (2025)
- [MASTEST: A LLM-Based Multi-Agent System For RESTful API Tests](https://mlsanity.com/paper/2511.18038/) (2025)

## Abstract (arXiv)

Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.