# ARTICLE: Annotator Reliability Through In-Context Learning

arXiv: https://arxiv.org/abs/2409.12218 · HTML: https://mlsanity.com/paper/2409.12218/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification](https://mlsanity.com/paper/2301.11562/) (2023)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://mlsanity.com/paper/2209.03392/) (2022)
- [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018)
- [Learning from Crowds with Crowd-Kit](https://mlsanity.com/paper/2109.08584/) (2021)
- [Incorporating Worker Perspectives into MTurk Annotation Practices for NLP](https://mlsanity.com/paper/2311.02802/) (2023)
- [Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive](https://mlsanity.com/paper/2301.12534/) (2023)

## Built on by (most cited first, 6 of 6)

- [STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems](https://mlsanity.com/paper/2605.02122/) (2026)
- [Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech](https://mlsanity.com/paper/2502.09004/) (2025)
- [DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning](https://mlsanity.com/paper/2509.09524/) (2025)
- [ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification](https://mlsanity.com/paper/2510.12534/) (2025)
- [Label Curation Using Agentic AI](https://mlsanity.com/paper/2602.02564/) (2026)
- [Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation](https://mlsanity.com/paper/2604.09626/) (2026)

## Abstract (arXiv)

Ensuring annotator quality in training and evaluation data is a key piece of machine learning in NLP. Tasks such as sentiment analysis and offensive speech detection are intrinsically subjective, creating a challenging scenario for traditional quality assessment approaches because it is hard to distinguish disagreement due to poor work from that due to differences of opinions between sincere annotators. With the goal of increasing diverse perspectives in annotation while ensuring consistency, we propose textttARTICLE, an in-context learning (ICL) framework to estimate annotation quality through self-consistency. We evaluate this framework on two offensive speech datasets using multiple LLMs and compare its performance with traditional methods. Our findings indicate that textttARTICLE can be used as a robust method for identifying reliable annotators, hence improving data quality.