# Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

arXiv: https://arxiv.org/abs/2601.16800 · HTML: https://mlsanity.com/paper/2601.16800/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [Knowing What, How and Why: A Near Complete Solution for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1911.01616/) (2019)
- [A Unified Generative Framework for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2106.04300/) (2021)
- [Position-Aware Tagging for Aspect Sentiment Triplet Extraction](https://mlsanity.com/paper/2010.02609/) (2020)
- [LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models](https://mlsanity.com/paper/2404.05221/) (2024)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Are Large Language Models Reliable Argument Quality Annotators?](https://mlsanity.com/paper/2404.09696/) (2024)
- [Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks](https://mlsanity.com/paper/2503.16974/) (2025)
- [Large Language Models for Propaganda Span Annotation](https://mlsanity.com/paper/2311.09812/) (2023)
- [LLMs as Span Annotators: A Comparative Study of LLMs and Humans](https://mlsanity.com/paper/2504.08697/) (2025)
- [If you've got it, flaunt it: Making the most of fine-grained sentiment annotations](https://mlsanity.com/paper/2102.00299/) (2021)

## Abstract (arXiv)

Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments and their targets. Although this level of detail is valuable, annotating opinions in datasets for model training requires considerable human effort and substantial cost, especially across diverse domains and real-world applications. To address this shortage of domain-specific labelled datasets, we explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis. We use a declarative annotation pipeline, an approach that reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a dedicated methodology for an LLM to adjudicate multiple labels and produce final annotations, benchmarked against exact, flexible, and element-wise variants of a rule-based voting aggregator. We trial the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks. Our results reveal a critical performance bifurcation: LLMs are reliable at the span level yet struggle to reproduce the relational structures that connect those spans faithfully. This suggests that LLMs are better positioned as high-fidelity annotation assistants and data augmentation tools to expand fine-grained opinion-annotated datasets, rather than replacing human annotators entirely.