# Generative AI and Empirical Software Engineering: A Paradigm Shift

arXiv: https://arxiv.org/abs/2502.08108 · HTML: https://mlsanity.com/paper/2502.08108/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT](https://mlsanity.com/paper/2308.04838/) (2023)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineering](https://mlsanity.com/paper/2501.08774/) (2025)
- [She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Models](https://mlsanity.com/paper/2303.10131/) (2023)
- [DevGPT: Studying Developer-ChatGPT Conversations](https://mlsanity.com/paper/2309.03914/) (2023)
- [From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs](https://mlsanity.com/paper/2404.11376/) (2024)
- [Scamming the Scammers: Using ChatGPT to Reply Mails for Wasting Time and Resources](https://mlsanity.com/paper/2303.13521/) (2023)
- [Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?](https://mlsanity.com/paper/2311.11081/) (2023)
- [Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace](https://mlsanity.com/paper/2410.18334/) (2024)
- [Are We Ready to Embrace Generative AI for Software Q&A?](https://mlsanity.com/paper/2307.09765/) (2023)
- [BugListener: Identifying and Synthesizing Bug Reports from Collaborative Live Chats](https://mlsanity.com/paper/2204.09368/) (2022)
- [A Disruptive Research Playbook for Studying Disruptive Innovations](https://mlsanity.com/paper/2402.13329/) (2024)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [Software Artifact Mining in Software Engineering Conferences: A Meta-Analysis](https://mlsanity.com/paper/2207.08436/) (2022)
- [Encoding Version History Context for Better Code Representation](https://mlsanity.com/paper/2402.03773/) (2024)
- [Bot-Driven Development: From Simple Automation to Autonomous Software Development Bots](https://mlsanity.com/paper/2411.16100/) (2024)
- [Software development in the age of LLMs and XR](https://mlsanity.com/paper/2404.09789/) (2024)

## Built on by (most cited first, 9 of 9)

- [Generating Energy-Efficient Code via Large-Language Models -- Where are we now?](https://mlsanity.com/paper/2509.10099/) (2025)
- [Exploiting LLM Agent Supply Chains via Payload-less Skills](https://mlsanity.com/paper/2605.14460/) (2026)
- [SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs](https://mlsanity.com/paper/2512.09543/) (2025)
- [Interacting with AI Reasoning Models: Harnessing "Thoughts" for AI-Driven Software Engineering](https://mlsanity.com/paper/2503.00483/) (2025)
- [Artificial Intelligence in Open Source Software Engineering: A Foundation for Sustainability](https://mlsanity.com/paper/2602.07071/) (2026)
- [Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs](https://mlsanity.com/paper/2603.27524/) (2026)
- [Enhancing Neural Code Representation with Additional Context](https://mlsanity.com/paper/2510.12082/) (2025)
- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)
- [Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers](https://mlsanity.com/paper/2608.27927/) (2026)

## Abstract (arXiv)

The adoption of large language models (LLMs) and autonomous agents in software engineering marks an enduring paradigm shift. These systems create new opportunities for tool design, workflow orchestration, and empirical observation, while fundamentally reshaping the roles of developers and the artifacts they produce. Although traditional empirical methods remain central to software engineering research, the rapid evolution of AI introduces new data modalities, alters causal assumptions, and challenges foundational constructs such as "developer", "artifact", and "interaction". As humans and AI agents increasingly co-create, the boundaries between social and technical actors blur, and the reproducibility of findings becomes contingent on model updates and prompt contexts. This vision paper examines how the integration of LLMs into software engineering disrupts established research paradigms. We discuss how it transforms the phenomena we study, the methods and theories we rely on, the data we analyze, and the threats to validity that arise in dynamic AI-mediated environments. Our aim is to help the empirical software engineering community adapt its questions, instruments, and validation standards to a future in which AI systems are not merely tools, but active collaborators shaping software engineering and its study.