# Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

arXiv: https://arxiv.org/abs/2408.05534 · HTML: https://mlsanity.com/paper/2408.05534/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 71 references that are in the map.
- First posted on arXiv 10 Aug 2024.

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)

## Built on by (most cited first, 30 of 42)

- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code](https://mlsanity.com/paper/2509.14744/) (2025)
- [Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Tasks](https://mlsanity.com/paper/2503.12029/) (2025)
- [Generative AI and Empirical Software Engineering: A Paradigm Shift](https://mlsanity.com/paper/2502.08108/) (2025)
- [Beyond the Prompt: An Empirical Study of Cursor Rules](https://mlsanity.com/paper/2512.18925/) (2025)
- [A Little Human Data Goes A Long Way](https://mlsanity.com/paper/2410.13098/) (2024)
- [Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks](https://mlsanity.com/paper/2504.01850/) (2025)
- [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026)
- [From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills](https://mlsanity.com/paper/2607.01456/) (2026)
- [Requirements Elicitation Follow-Up Question Generation](https://mlsanity.com/paper/2507.02858/) (2025)
- [Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection](https://mlsanity.com/paper/2505.20209/) (2025)
- [Delving into Instance-Dependent Label Noise in Graph Data: A Comprehensive Study and Benchmark](https://mlsanity.com/paper/2506.12468/) (2025)
- [How Do Agents Perform Code Optimization? An Empirical Study](https://mlsanity.com/paper/2512.21757/) (2025)
- [Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research](https://mlsanity.com/paper/2506.12691/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [Human-AI Synergy in Agentic Code Review](https://mlsanity.com/paper/2603.15911/) (2026)
- [A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics](https://mlsanity.com/paper/2505.15469/) (2025)
- [Evaluating the Use of LLMs for Documentation to Code Traceability](https://mlsanity.com/paper/2506.16440/) (2025)
- [AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research](https://mlsanity.com/paper/2507.08038/) (2025)
- [RUBAS: Rubric-Based Reinforcement Learning for Agent Safety](https://mlsanity.com/paper/2606.04051/) (2026)
- [From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained](https://mlsanity.com/paper/2607.00911/) (2026)
- [Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement](https://mlsanity.com/paper/2505.20973/) (2025)
- [OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering](https://mlsanity.com/paper/2512.15979/) (2025)
- [Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks](https://mlsanity.com/paper/2604.05100/) (2026)
- [What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems](https://mlsanity.com/paper/2605.20548/) (2026)
- [An Empirical Study of Downstream Adaptation for Agent Skills](https://mlsanity.com/paper/2607.03238/) (2026)
- [Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates](https://mlsanity.com/paper/2608.29455/) (2026)
- [AI-Mediated Code Comment Improvement](https://mlsanity.com/paper/2505.09021/) (2025)

## Abstract (arXiv)

Experimental evaluations of software engineering innovations, e.g., tools and processes, often include human-subject studies as a component of a multi-pronged strategy to obtain greater generalizability of the findings. However, human-subject studies in our field are challenging, due to the cost and difficulty of finding and employing suitable subjects, ideally, professional programmers with varying degrees of experience. Meanwhile, large language models (LLMs) have recently started to demonstrate human-level performance in several areas. This paper explores the possibility of substituting costly human subjects with much cheaper LLM queries in evaluations of code and code-related artifacts. We study this idea by applying six state-of-the-art LLMs to ten annotation tasks from five datasets created by prior work, such as judging the accuracy of a natural language summary of a method or deciding whether a code change fixes a static analysis warning. Our results show that replacing some human annotation effort with LLMs can produce inter-rater agreements equal or close to human-rater agreement. To help decide when and how to use LLMs in human-subject studies, we propose model-model agreement as a predictor of whether a given task is suitable for LLMs at all, and model confidence as a means to select specific samples where LLMs can safely replace human annotators. Overall, our work is the first step toward mixed human-LLM evaluations in software engineering.