mlsanityData to 7 Oct 2026

The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators

arXiv 2407.11004 ↗First posted 25 Jun 2024cs.CL cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, Alchemist, obtains comparable to or better performance than large language model-based annotation in a range of tasks for a fraction of the cost: on average, improvements amount to a 12.9% enhancement while the total labeling costs across all datasets are reduced by a factor of approximately 500x.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

39 of 56 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
27 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  2. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  3. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  4. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  5. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  6. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  7. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  8. 2022In-context Examples Selection for Machine Translationcited by 172
  9. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  10. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  11. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 104
  12. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  13. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  14. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  15. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  16. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  17. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  18. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  19. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  20. 2020Denoising Multi-Source Weak Supervision for Neural Text Classificationcited by 42
  21. 2022Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approachescited by 39
  22. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  23. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  24. 2019GOGGLES: Automatic Image Labeling with Affinity Codingcited by 19
  25. 2022AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labelscited by 7
  26. 2023Can Large Language Models Design Accurate Label Functions?cited by 6
  27. 2023GPT Self-Supervision for a Better Data Annotatorcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2024LLM Chain Ensembles for Scalable and Accurate Data Annotationcited by 11
  2. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 10
  3. 2025Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluationcited by 5
  4. 2025Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weightscited by 3
  5. 2025Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detectioncited by 2
  6. 2026Structured Exploration and Exploitation of Label Functions for Automated Data Annotationcited by 1
  7. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  8. 2026The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegramcited by 0
  9. 2026Noise-Aware Framework for Correcting Corrupted Labelscited by 0
  10. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0