# Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study

arXiv: https://arxiv.org/abs/2404.12772 · HTML: https://mlsanity.com/paper/2404.12772/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 10 of its 39 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs](https://mlsanity.com/paper/2310.13976/) (2023)
- [Generative Artificial Intelligence for Software Engineering -- A Research Agenda](https://mlsanity.com/paper/2310.18648/) (2023)
- [A Taxonomy for Requirements Engineering and Software Test Alignment](https://mlsanity.com/paper/2307.12477/) (2023)

## Built on by (most cited first, 9 of 9)

- [Generative AI for Requirements Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2409.06741/) (2024)
- [Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Models](https://mlsanity.com/paper/2507.05981/) (2025)
- [RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice](https://mlsanity.com/paper/2502.14930/) (2025)
- [LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems](https://mlsanity.com/paper/2501.11864/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [From Domain Documents to Requirements: Retrieval-Augmented Generation in the Space Industry](https://mlsanity.com/paper/2507.07689/) (2025)
- [AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps](https://mlsanity.com/paper/2606.06563/) (2026)
- [Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generation](https://mlsanity.com/paper/2602.04726/) (2026)
- [WebMAC: A Multi-Agent Collaborative Framework for Scenario Testing of Web Systems](https://mlsanity.com/paper/2604.13559/) (2026)

## Abstract (arXiv)

Test scenarios are specific instances of test cases that describe actions to validate a particular software functionality. By outlining the conditions under which the software operates and the expected outcomes, test scenarios ensure that the software functionality is tested in an integrated manner. Test scenarios are crucial for systematically testing an application under various conditions, including edge cases, to identify potential issues and guarantee overall performance and reliability. Specifying test scenarios is tedious and requires a deep understanding of software functionality and the underlying domain. It further demands substantial effort and investment from already time- and budget-constrained requirements engineers and testing teams. This paper presents an automated approach (RAGTAG) for test scenario generation using Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs). RAG allows the integration of specific domain knowledge with LLMs' generation capabilities. We evaluate RAGTAG on two industrial projects from Austrian Post with bilingual requirements in German and English. Our results from an interview survey conducted with four experts on five dimensions -- relevance, coverage, correctness, coherence and feasibility, affirm the potential of RAGTAG in automating test scenario generation. Specifically, our results indicate that, despite the difficult task of analyzing bilingual requirements, RAGTAG is able to produce scenarios that are well-aligned with the underlying requirements and provide coverage of different aspects of the intended functionality. The generated scenarios are easily understandable to experts and feasible for testing in the project environment. The overall correctness is deemed satisfactory; however, gaps in capturing exact action sequences and domain nuances remain, underscoring the need for domain expertise when applying LLMs.