# Guidelines for including grey literature and conducting multivocal literature reviews in software engineering

arXiv: https://arxiv.org/abs/1707.02553 · HTML: https://mlsanity.com/paper/1707.02553/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2017.

## Built on by (most cited first, 30 of 41)

- [AgentOps: Enabling Observability of LLM Agents](https://mlsanity.com/paper/2411.05285/) (2024)
- [Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering](https://mlsanity.com/paper/2209.04963/) (2022)
- [Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems](https://mlsanity.com/paper/2103.14101/) (2021)
- [Generative AI for Software Architecture. Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2503.13310/) (2025)
- [The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice](https://mlsanity.com/paper/2301.09001/) (2023)
- [Differential Privacy in Natural Language Processing: The Story So Far](https://mlsanity.com/paper/2208.08140/) (2022)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [A Taxonomy of Foundation Model based Systems through the Lens of Software Architecture](https://mlsanity.com/paper/2305.05352/) (2023)
- [Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review](https://mlsanity.com/paper/2409.11864/) (2024)
- [The impact of using biased performance metrics on software defect prediction research](https://mlsanity.com/paper/2103.10201/) (2021)
- [Eliciting Best Practices for Collaboration with Computational Notebooks](https://mlsanity.com/paper/2202.07233/) (2022)
- [Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems](https://mlsanity.com/paper/2203.10384/) (2022)
- [Design Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository](https://mlsanity.com/paper/2303.13173/) (2023)
- [Exploring Agentic Artificial Intelligence Systems: Towards a Typological Framework](https://mlsanity.com/paper/2508.00844/) (2025)
- [Agentic AI in 6G Software Businesses: A Layered Maturity Model](https://mlsanity.com/paper/2508.03393/) (2025)
- [LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review](https://mlsanity.com/paper/2604.16321/) (2026)
- [Responsible AI Question Bank: A Comprehensive Tool for AI Risk Assessment](https://mlsanity.com/paper/2408.11820/) (2024)
- ["We are not Future-ready": Understanding AI Privacy Risks and Existing Mitigation Strategies from the Perspective of AI Developers in Europe](https://mlsanity.com/paper/2510.00909/) (2025)
- [Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Education](https://mlsanity.com/paper/2512.23982/) (2025)
- [Assessing the Quality of Computational Notebooks for a Frictionless Transition from Exploration to Production](https://mlsanity.com/paper/2205.11941/) (2022)
- [A Survey Study on the State of the Art of Programming Exercise Generation using Large Language Models](https://mlsanity.com/paper/2405.20183/) (2024)
- [Combating Toxic Language: A Review of LLM-Based Strategies for Software Engineering](https://mlsanity.com/paper/2504.15439/) (2025)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Configuration Smells in AGENTS.md Files: Common Mistakes in Configuring Coding Agents](https://mlsanity.com/paper/2606.15828/) (2026)
- [Harnessing Agent Skills: Architectural Patterns and a Reference Architecture for Skill-Mediated LLM Agents](https://mlsanity.com/paper/2606.20631/) (2026)
- [SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks](https://mlsanity.com/paper/2608.01117/) (2026)
- [Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report](https://mlsanity.com/paper/2608.11965/) (2026)
- [Good Tools are Half the Work: Tool Usage in Deep Learning Projects](https://mlsanity.com/paper/2310.19124/) (2023)
- [Towards a Small Language Model Lifecycle Framework](https://mlsanity.com/paper/2506.07695/) (2025)
- [A Multi-Level Strategy for Deepfake Content Moderation under EU Regulation](https://mlsanity.com/paper/2507.08879/) (2025)

## Abstract (arXiv)

Context: A Multivocal Literature Review (MLR) is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). MLRs are useful for both researchers and practitioners since they provide summaries both the state-of-the art and -practice in a given area. Objective: There are several guidelines to conduct SLR studies in SE. However, given the facts that several phases of MLRs differ from those of traditional SLRs, for instance with respect to the search process and source quality assessment. Therefore, SLR guidelines are only partially useful for conducting MLR studies. Our goal in this paper is to present guidelines on how to conduct MLR studies in SE. Method: To develop the MLR guidelines, we benefit from three inputs: (1) existing SLR guidelines in SE, (2), a literature survey of MLR guidelines and experience papers in other fields, and (3) our own experiences in conducting several MLRs in SE. All derived guidelines are discussed in the context of three examples MLRs as running examples (two from SE and one MLR from the medical sciences). Results: The resulting guidelines cover all phases of conducting and reporting MLRs in SE from the planning phase, over conducting the review to the final reporting of the review. In particular, we believe that incorporating and adopting a vast set of recommendations from MLR guidelines and experience papers in other fields have enabled us to propose a set of guidelines with solid foundations. Conclusion: Having been developed on the basis of three types of solid experience and evidence, the provided MLR guidelines support researchers to effectively and efficiently conduct new MLRs in any area of SE.