LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine
Not on the map. Why?
Abstract (arXiv)
The Cancer Registry of Norway (CRN) uses an automated cancer registration support system (CaReSS) to support core cancer registry activities, i.e, data capture, data curation, and producing data products and statistics for various stakeholders. GURI is a core component of CaReSS, which is responsible for validating incoming data with medical rules. Such medical rules are manually implemented by medical experts based on medical standards, regulations, and research. Since large language models (LLMs) have been trained on a large amount of public information, including these documents, they can be employed to generate tests for GURI. Thus, we propose an LLM-based test generation and differential testing approach (LLMeDiff) to test GURI. We experimented with four different LLMs, two medical rule engine implementations, and 58 real medical rules to investigate the hallucination, success, time efficiency, and robustness of the LLMs to generate tests, and these tests' ability to find potential issues in GURI. Our results showed that GPT-3.5 hallucinates the least, is the most successful, and is generally the most robust; however, it has the worst time efficiency. Our differential testing revealed 22 medical rules where implementation inconsistencies were discovered (e.g., regarding handling rule versions). Finally, we provide insights for practitioners and researchers based on the results.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 62 references that are in the map. First posted on arXiv 16 Feb 2024.
Builds on
24 of 62 references.
References · 24
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Mistral 7Bcited by 3,164
- 2024Mixtral of Expertscited by 1,823
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2022A Systematic Evaluation of Large Language Models of Codecited by 363
- 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 166
- 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
- 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
- 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
- 2023Cumulative Reasoning with Large Language Modelscited by 94
- 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
12 more
- 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 54
- 2023Effective Test Generation Using Pre-trained Large Language Models and Mutation Testingcited by 51
- 2023What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?cited by 47
- 2023Can Large Language Models Write Good Property-Based Tests?cited by 31
- 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
- 2023Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binariescited by 26
- 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
- 2023EvoCLINICAL: Evolving Cyber-Cyber Digital Twin with Active Transfer Learning for Automated Cancer Registry Systemcited by 3
- 2023Automated Test Generation for Medical Rules Web Services: A Case Study at the Cancer Registry of Norwaycited by 2
- 2023Challenges of Testing an Evolving Cancer Registration Support System in Practicecited by 2
- 2023Cost Reduction on Testing Evolving Cancer Registry Systemcited by 2
- 2023Large Language Models in Medical Term Classification and Unexpected Misalignment Between Response and Reasoningcited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.