# Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation

arXiv: https://arxiv.org/abs/2510.25904 · HTML: https://mlsanity.com/paper/2510.25904/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold](https://mlsanity.com/paper/1706.09528/) (2017)
- [LOME: Large Ontology Multilingual Extraction](https://mlsanity.com/paper/2101.12175/) (2021)
- [(Re)construing Meaning in NLP](https://mlsanity.com/paper/2005.09099/) (2020)
- [Open-Domain Frame Semantic Parsing Using Transformers](https://mlsanity.com/paper/2010.10998/) (2020)
- [Coarse-to-Fine Dual Encoders are Better Frame Identification Learners](https://mlsanity.com/paper/2310.13316/) (2023)
- [Can LLMs Extract Frame-Semantic Arguments?](https://mlsanity.com/paper/2502.12516/) (2025)
- [Do LLMs Encode Frame Semantics? Evidence from Frame Identification](https://mlsanity.com/paper/2509.19540/) (2025)
- [Domain Adaptation in Neural Machine Translation using a Qualia-Enriched FrameNet](https://mlsanity.com/paper/2202.10287/) (2022)
- [Annotating FrameNet via Structure-Conditioned Language Generation](https://mlsanity.com/paper/2406.04834/) (2024)
- [Exploring In-Context Learning for Frame-Semantic Parsing](https://mlsanity.com/paper/2507.23082/) (2025)

## Abstract (arXiv)

The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time.