# Sampling in Software Engineering Research: A Critical Review and Guidelines

arXiv: https://arxiv.org/abs/2002.07764 · HTML: https://mlsanity.com/paper/2002.07764/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 23 of 23)

- [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025)
- [Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers](https://mlsanity.com/paper/2403.15600/) (2024)
- [SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation](https://mlsanity.com/paper/2507.09108/) (2025)
- [Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations](https://mlsanity.com/paper/2407.08983/) (2024)
- [Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical Study](https://mlsanity.com/paper/2401.11105/) (2024)
- [Ethics in the Age of AI: An Analysis of AI Practitioners' Awareness and Challenges](https://mlsanity.com/paper/2307.10057/) (2023)
- [TEP-GNN: Accurate Execution Time Prediction of Functional Tests using Graph Neural Networks](https://mlsanity.com/paper/2208.11947/) (2022)
- [CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models](https://mlsanity.com/paper/2503.16167/) (2025)
- [From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained](https://mlsanity.com/paper/2607.00911/) (2026)
- [The Perspective of Software Professionals on Algorithmic Racism](https://mlsanity.com/paper/2306.15133/) (2023)
- [Navigating Fairness: Practitioners' Understanding, Challenges, and Strategies in AI/ML Development](https://mlsanity.com/paper/2403.15481/) (2024)
- [Enabling Global, Human-Centered Explanations for LLMs:From Tokens to Interpretable Code and Test Generation](https://mlsanity.com/paper/2503.16771/) (2025)
- [How Do Companies Manage the Environmental Sustainability of AI? An Interview Study About Green AI Efforts and Regulations](https://mlsanity.com/paper/2505.07317/) (2025)
- [A Taxonomy of Runtime Faults in Model Context Protocol Servers](https://mlsanity.com/paper/2606.05339/) (2026)
- [Pattern Discovery and Validation Using Scientific Research Methods](https://mlsanity.com/paper/2107.06065/) (2021)
- [Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectives](https://mlsanity.com/paper/2503.13793/) (2025)
- [A Survey for What Developers Require in AI-powered Tools that Aid in Component Selection in CBSD](https://mlsanity.com/paper/2504.13751/) (2025)
- [Integrating Human Feedback into a Reinforcement Learning-Based Framework for Adaptive User Interfaces](https://mlsanity.com/paper/2504.20782/) (2025)
- [Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Software](https://mlsanity.com/paper/2510.15494/) (2025)
- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)
- [ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems](https://mlsanity.com/paper/2606.21597/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests](https://mlsanity.com/paper/2608.02685/) (2026)

## Abstract (arXiv)

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.