mlsanityData to 7 Oct 2026

LLM Chain Ensembles for Scalable and Accurate Data Annotation

arXiv 2410.13006 ↗First posted 16 Oct 2024cs.LG cs.SIcited by 11

Not on the map. Why?

Abstract (arXiv)

The ability of large language models (LLMs) to perform zero-shot classification makes them viable solutions for data annotation in rapidly evolving domains where quality labeled data is often scarce and costly to obtain. However, the large-scale deployment of LLMs can be prohibitively expensive. This paper introduces an LLM chain ensemble methodology that aligns multiple LLMs in a sequence, routing data subsets to subsequent models based on classification uncertainty. This approach leverages the strengths of individual LLMs within a broader system, allowing each model to handle data points where it exhibits the highest confidence, while forwarding more complex cases to potentially more robust models. Our results show that the chain ensemble method often exceeds the performance of the best individual model in the chain and achieves substantial cost savings, making LLM chain ensembles a practical and efficient solution for large-scale data annotation challenges.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

25 of 34 references.

References · 25

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  6. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  8. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  9. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  10. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  11. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  12. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
13 more
  1. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  2. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  3. 2023Stance Detection with Collaborative Role-Infused LLM-Based Agentscited by 53
  4. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  5. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  6. 2021Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlinescited by 32
  7. 2023Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classificationcited by 16
  8. 2023Chain-of-Thought Embeddings for Stance Detection on Social Mediacited by 13
  9. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  10. 2024LLMs' Classification Performance is Overclaimedcited by 6
  11. 2024Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performancecited by 5
  12. 2024Chain of Stance: Stance Detection with Large Language Modelscited by 5
  13. 2024RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Sciencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2025Labeling Free-text Data using Language Model Ensemblescited by 5
  2. 2024Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Languagecited by 4
  3. 2025MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platformcited by 3
  4. 2025What to Keep and What to Drop: Adaptive Table Filtering Frameworkcited by 1
  5. 2025Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scalecited by 1
  6. 2025That's So FETCH: Fashioning Ensemble Techniques for LLM Classification in Civil Legal Intake and Referralcited by 1
  7. 2026D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximationcited by 1
  8. 2024Sparse Linear Bandits with Blocking Constraintscited by 0
  9. 2025Local LLM Ensembles for Zero-shot Portuguese Named Entity Recognitioncited by 0
  10. 2026Temporal Narrative Monitoring in Dynamic Information Environmentscited by 0
  11. 2026Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Modelscited by 0