mlsanityData to 7 Oct 2026

Generative AI and Empirical Software Engineering: A Paradigm Shift

arXiv 2502.08108 ↗First posted 12 Feb 2025cs.SE cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

The adoption of large language models (LLMs) and autonomous agents in software engineering marks an enduring paradigm shift. These systems create new opportunities for tool design, workflow orchestration, and empirical observation, while fundamentally reshaping the roles of developers and the artifacts they produce. Although traditional empirical methods remain central to software engineering research, the rapid evolution of AI introduces new data modalities, alters causal assumptions, and challenges foundational constructs such as "developer", "artifact", and "interaction". As humans and AI agents increasingly co-create, the boundaries between social and technical actors blur, and the reproducibility of findings becomes contingent on model updates and prompt contexts. This vision paper examines how the integration of LLMs into software engineering disrupts established research paradigms. We discuss how it transforms the phenomena we study, the methods and theories we rely on, the data we analyze, and the threats to validity that arise in dynamic AI-mediated environments. Our aim is to help the empirical software engineering community adapt its questions, instruments, and validation standards to a future in which AI systems are not merely tools, but active collaborators shaping software engineering and its study.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 23 of its 56 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

23 of 56 references.

References · 23

  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  3. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  4. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  5. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  6. 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 45
  7. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  8. 2024The Future of AI-Driven Software Engineeringcited by 23
  9. 2025How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineeringcited by 14
  10. 2023She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Modelscited by 12
  11. 2023DevGPT: Studying Developer-ChatGPT Conversationscited by 11
  12. 2024From Image to UML: First Results of Image Based UML Diagram Generation Using LLMscited by 11
11 more
  1. 2023Scamming the Scammers: Using ChatGPT to Reply Mails for Wasting Time and Resourcescited by 8
  2. 2023Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?cited by 8
  3. 2024Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplacecited by 7
  4. 2023Are We Ready to Embrace Generative AI for Software Q&A?cited by 5
  5. 2022BugListener: Identifying and Synthesizing Bug Reports from Collaborative Live Chatscited by 4
  6. 2024A Disruptive Research Playbook for Studying Disruptive Innovationscited by 4
  7. 2024Leveraging Reviewer Experience in Code Review Comment Generationcited by 4
  8. 2022Software Artifact Mining in Software Engineering Conferences: A Meta-Analysiscited by 2
  9. 2024Encoding Version History Context for Better Code Representationcited by 2
  10. 2024Bot-Driven Development: From Simple Automation to Autonomous Software Development Botscited by 2
  11. 2024Software development in the age of LLMs and XRcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.