mlsanityData to 7 Oct 2026

The Importance of Human-Labeled Data in the Era of LLMs

arXiv 2306.14910 ↗First posted 18 Jun 2023cs.CL cs.AI cs.CYcited by 8

Not on the map. Why?

Abstract (arXiv)

The advent of large language models (LLMs) has brought about a revolution in the development of tailored machine learning models and sparked debates on redefining data requirements. The automation facilitated by the training and implementation of LLMs has led to discussions and aspirations that human-level labeling interventions may no longer hold the same level of importance as in the era of supervised learning. This paper presents compelling arguments supporting the ongoing relevance of human-labeled data in the era of LLMs.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 41 of its 76 references that are in the map. First posted on arXiv 18 Jun 2023.

Builds on

41 of 76 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2018Model Cards for Model Reportingcited by 1,068
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
29 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2020Performative Predictioncited by 318
  4. 2015Strategic Classificationcited by 256
  5. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  6. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  7. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  8. 2020A Second-Order Approach to Learning with Instance-Dependent Label Noisecited by 90
  9. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  10. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  11. 2020How Do Fair Decisions Fare in Long-term Qualification?cited by 59
  12. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 57
  13. 2019How model accuracy and explanation fidelity influence user trustcited by 44
  14. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  15. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  16. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  17. 2022Fairness Transferability Subject to Bounded Distribution Shiftcited by 28
  18. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  19. 2023Long-Term Fairness with Unknown Dynamicscited by 25
  20. 2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24
  21. 2021Understanding Instance-Level Label Noise: Disparate Impacts and Treatmentscited by 22
  22. 2021Unintended Selection: Persistent Qualification Rate Disparities and Interventionscited by 18
  23. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  24. 2014Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcingcited by 11
  25. 2021Model Transferability With Responsive Decision Subjectscited by 9
  26. 2021Unfairness Despite Awareness: Group-Fair Classification with Strategic Agentscited by 9
  27. 2023Human and AI Perceptual Differences in Image Classification Errorscited by 9
  28. 2016Learning to Incentivize: Eliciting Effort via Output Agreementcited by 8
  29. 2020Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replicationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 1.

8 papers.

  1. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 32
  2. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  3. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  4. 2024LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domainscited by 7
  5. 2024Rethinking Software Engineering in the Foundation Model Era: A Curated Catalogue of Challenges in the Development of Trustworthy FMwarecited by 6
  6. 2025Structured Extraction of Process Structure Properties Relationships in Materials Sciencecited by 1
  7. 2025Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspectivecited by 1
  8. 2026Enhancing Reinforcement Learning Fine-Tuning with an Online Refinercited by 0