Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research
Not on the map. Why?
Abstract (arXiv)
The adoption of Large Language Models (LLMs) is not only transforming software engineering (SE) practice but is also poised to fundamentally disrupt how research is conducted in the field. While perspectives on this transformation range from viewing LLMs as mere productivity tools to considering them revolutionary forces, we argue that the SE research community must proactively engage with and shape the integration of LLMs into research practices, emphasizing human agency in this transformation. As LLMs rapidly become integral to SE research - both as tools that support investigations and as subjects of study - a human-centric perspective is essential. Ensuring human oversight and interpretability is necessary for upholding scientific rigor, fostering ethical responsibility, and driving advancements in the field. Drawing from discussions at the 2nd Copenhagen Symposium on Human-Centered AI in SE, this position paper employs McLuhan's Tetrad of Media Laws to analyze the impact of LLMs on SE research. Through this theoretical lens, we examine how LLMs enhance research capabilities through accelerated ideation and automated processes, make some traditional research practices obsolete, retrieve valuable aspects of historical research approaches, and risk reversal effects when taken to extremes. Our analysis reveals opportunities for innovation and potential pitfalls that require careful consideration. We conclude with a call to action for the SE research community to proactively harness the benefits of LLMs while developing frameworks and guidelines to mitigate their risks, to ensure continued rigor and impact of research in an AI-augmented future.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 11 of its 29 references that are in the map. First posted on arXiv 15 Jun 2025.
Builds on
11 of 29 references.
References · 11
- 2024AI-Augmented Brainwriting: Investigating the use of LLMs in group ideationcited by 44
- 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 42
- 2023Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languagescited by 36
- 2024Towards Evaluation Guidelines for Empirical Studies involving LLMscited by 15
- 2023Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Modelscited by 13
- 2024DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Explorationcited by 10
- 2023Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?cited by 8
- 2020Empirical Standards for Software Engineering Researchcited by 4
- 2024A Disruptive Research Playbook for Studying Disruptive Innovationscited by 4
- 2024Guiding Principles for Using Mixed Methods Research in Software Engineeringcited by 4
- 2022Social Science Theories in Software Engineering Researchcited by 1
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
- 2025Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligencecited by 6
- 2025A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AIcited by 1
- 2026Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researcherscited by 0
- 2026Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers28 Aug 2026
- 2025A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AI30 Oct 2025
- 2025Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence28 Sep 2025