mlsanityData to 7 Oct 2026

Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets

arXiv 2309.06503 ↗First posted 12 Sep 2023cs.CL cs.LG cs.SIcited by 8

Not on the map. Why?

Abstract (arXiv)

The COVID-19 pandemic has presented significant challenges to the healthcare industry and society as a whole. With the rapid development of COVID-19 vaccines, social media platforms have become a popular medium for discussions on vaccine-related topics. Identifying vaccine-related tweets and analyzing them can provide valuable insights for public health research-ers and policymakers. However, manual annotation of a large number of tweets is time-consuming and expensive. In this study, we evaluate the usage of Large Language Models, in this case GPT-4 (March 23 version), and weak supervision, to identify COVID-19 vaccine-related tweets, with the purpose of comparing performance against human annotators. We leveraged a manu-ally curated gold-standard dataset and used GPT-4 to provide labels without any additional fine-tuning or instructing, in a single-shot mode (no additional prompting).

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 35 of its 67 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

35 of 67 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
23 more
  1. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  2. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  5. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  6. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  7. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  8. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  9. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  10. 2020COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twittercited by 96
  11. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  12. 2022A Survey on Programmatic Weak Supervisioncited by 66
  13. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  14. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  15. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  16. 2020A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaborationcited by 33
  17. 2021Universalizing Weak Supervisioncited by 27
  18. 2020Strength from Weakness: Fast Learning Using Weak Supervisioncited by 18
  19. 2019Active Learning with Point Supervision for Cost-Effective Panicle Detection in Cereal Cropscited by 8
  20. 2017Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classificationcited by 4
  21. 2022TweetDIS: A Large Twitter Dataset for Natural Disasters Built using Weak Supervisioncited by 2
  22. 2022Label Augmentation with Reinforced Labeling for Weak Supervisioncited by 1
  23. 2022Identifying epidemic related Tweets using noisy learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labelingcited by 8
  2. 2024Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Taskscited by 3
  3. 2024GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Textcited by 3
  4. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
  5. 2024Scope of Large Language Models for Mining Emerging Opinions in Online Health Discoursecited by 2
  6. 2024Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysiscited by 2
  7. 2025MisinfoTeleGraph: Network-driven Misinformation Detection for German Telegram Messagescited by 1
  8. 2026Unsupervised Anomaly Detection of Information Operations Users via Behavioral and Language Patternscited by 0