mlsanityData to 14 Oct 2026

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

arXiv 2306.15766 ↗First posted 27 Jun 2023cs.CL cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of large language models (LLMs) for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

30 of 36 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  10. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  11. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  12. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
18 more
  1. 2022Is GPT-3 a Good Data Annotator?cited by 208
  2. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  3. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  4. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  5. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  6. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  7. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  8. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
  9. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  10. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  11. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  12. 2020Rethinking Distributional Matching Based Domain Adaptationcited by 33
  13. 2022NGAME: Negative Mining-aware Mini-batching for Extreme Classificationcited by 32
  14. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  15. 2020Domain Divergences: a Survey and Empirical Analysiscited by 24
  16. 2023Targeted Data Generation: Finding and Fixing Model Weaknessescited by 15
  17. 2023UDApter -- Efficient Domain Adaptation Using Adapterscited by 13
  18. 2022To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 14; 2025: 5; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  2. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  3. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  4. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  5. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  6. 2023Towards LLM-guided Causal Explainability for Black-box Text Classifierscited by 44
  7. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  8. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  9. 2023Large Language Models for Propaganda Span Annotationcited by 19
  10. 2025From Selection to Generation: A Survey of LLM-based Active Learningcited by 19
  11. 2024Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationalescited by 16
  12. 2024Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotationcited by 16
21 more
  1. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 12
  2. 2024GPTs Are Multilingual Annotators for Sequence Generation Taskscited by 11
  3. 2025Evaluating Large Language Models as Expert Annotatorscited by 9
  4. 2026Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMscited by 9
  5. 2023Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognitioncited by 5
  6. 2024GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Textcited by 4
  7. 2025Revisiting Active Learning under (Human) Label Variationcited by 4
  8. 2024VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Taskscited by 3
  9. 2024Are Expert-Level Language Models Expert-Level Annotators?cited by 3
  10. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  11. 2023It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Modelscited by 2
  12. 2024USDC: A Dataset of underlineUser underlineStance and underlineDogmatism in Long underlineConversationscited by 1
  13. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inferencecited by 1
  14. 2026Under-resourced studies of under-resourced languages: lemmatization and POS-tagging with LLM annotators for historical Armenian, Georgian, Greek and Syriaccited by 1
  15. 2026Labeling Training Data for Entity Matching Using Large Language Modelscited by 1
  16. 2026SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematicscited by 1
  17. 2024Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inferencecited by 0
  18. 2024Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciencescited by 0
  19. 2025Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsingcited by 0
  20. 2026MVSS: A Unified Framework for Multi-View Structured Survey Generationcited by 0
  21. 2026Occupational Prompting Reveals Cultural Bias in Large Language Modelscited by 0