# OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering

arXiv: https://arxiv.org/abs/2512.15979 · HTML: https://mlsanity.com/paper/2512.15979/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Get my drift? Catching LLM Task Drift with Activation Deltas](https://mlsanity.com/paper/2406.00799/) (2024)
- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [POSIX: A Prompt Sensitivity Index For Large Language Models](https://mlsanity.com/paper/2410.02185/) (2024)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation](https://mlsanity.com/paper/2404.09682/) (2024)
- [Leveraging LLMs for the Quality Assurance of Software Requirements](https://mlsanity.com/paper/2408.10886/) (2024)
- [Towards Evaluation Guidelines for Empirical Studies involving LLMs](https://mlsanity.com/paper/2411.07668/) (2024)
- [The State of Documentation Practices of Third-party Machine Learning Models and Datasets](https://mlsanity.com/paper/2312.15058/) (2023)
- [Aggregating Soft Labels from Crowd Annotations Improves Uncertainty Estimation Under Distribution Shift](https://mlsanity.com/paper/2212.09409/) (2022)

## Built on by (most cited first, 1 of 1)

- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in empirical software engineering (ESE) to automate or assist annotation tasks such as labeling commits, issues, and qualitative artifacts. Yet the reliability and reproducibility of such annotations remain underexplored. Existing studies often lack standardized measures for reliability, calibration, and drift, and frequently omit essential configuration details. We argue that LLM-based annotation should be treated as a measurement process rather than a purely automated activity. In this position paper, we outline the Operationalization for LLM-based Annotation Framework (OLAF), a conceptual framework that organizes key constructs: textitreliability, calibration, drift, consensus, aggregation, and textittransparency. The paper aims to motivate methodological discussion and future empirical work toward more transparent and reproducible LLM-based annotation in software engineering research.