# Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies

arXiv: https://arxiv.org/abs/2307.02612 · HTML: https://mlsanity.com/paper/2307.02612/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2023.

## Built on by (most cited first, 16 of 16)

- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Artificial Intelligence for Literature Reviews: Opportunities and Challenges](https://mlsanity.com/paper/2402.08565/) (2024)
- [A Critical Survey on Fairness Benefits of Explainable AI](https://mlsanity.com/paper/2310.13007/) (2023)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [Large Language Model for Qualitative Research -- A Systematic Mapping Study](https://mlsanity.com/paper/2411.14473/) (2024)
- [Generative AI for Testing of Autonomous Driving Systems: A Survey](https://mlsanity.com/paper/2508.19882/) (2025)
- [Aggregating empirical evidence from data strategy studies: a case on model quantization](https://mlsanity.com/paper/2505.00816/) (2025)
- [Agile Management for Machine Learning: A Systematic Mapping Study](https://mlsanity.com/paper/2506.20759/) (2025)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Hyperspectral Anomaly Detection Methods: A Survey and Comparative Study](https://mlsanity.com/paper/2507.05730/) (2025)
- [A Multi-Level Strategy for Deepfake Content Moderation under EU Regulation](https://mlsanity.com/paper/2507.08879/) (2025)
- [Self-organizing maps for water quality assessment in reservoirs and lakes: A systematic literature review](https://mlsanity.com/paper/2512.18466/) (2025)
- [On the Use of a Large Language Model to Support the Conduction of a Systematic Mapping Study: A Brief Report from a Practitioner's View](https://mlsanity.com/paper/2602.10147/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews](https://mlsanity.com/paper/2603.05514/) (2026)
- [Where are the Humans? A Scoping Review of Fairness in Multi-agent AI Systems](https://mlsanity.com/paper/2604.15078/) (2026)

## Abstract (arXiv)

Background: A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which has later been complemented with snowball searches. Our conjecture is that we can perform even better searches if combining the two search approaches, referred to as a hybrid search strategy. Objective: Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared some alternative hybrid search strategies to assess whether it was possible to identify more cost-efficient ways of searching for relevant primary studies. Method: To compare and evaluate the hybrid search strategy, we replicated an SLR on industry-academia collaboration in software engineering. The SLR used a more traditional approach to searching for relevant articles for an SLR, while the replication was conducted using a hybrid search strategy. Results: In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30 percent more primary studies and even more when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when conducting systematic literature studies. Conclusions: The hybrid search strategy is a strong contender for being used when conducting systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.