mlsanityData to 14 Oct 2026

ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

arXiv 2303.03953 ↗First posted 7 Mar 2023cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. In this paper, we examine whether ChatGPT can be used for zero-shot text classification, more specifically, automatic genre identification. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. The models are compared on test sets in two languages: English and Slovenian. Results show that ChatGPT outperforms the fine-tuned model when applied to the dataset which was not seen before by either of the models. Even when applied on Slovenian language as an under-resourced language, ChatGPT's performance is no worse than when applied to English. However, if the model is fully prompted in Slovenian, the performance drops significantly, showing the current limitations of ChatGPT usage on smaller languages. The presented results lead us to questioning whether this is the beginning of an end of laborious manual annotation campaigns even for smaller languages, such as Slovenian.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 16 of its 47 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

16 of 47 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  9. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  10. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  11. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  12. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
4 more
  1. 2022How would Stance Detection Techniques Evolve after the Launch of ChatGPT?cited by 76
  2. 2021Genre as Weak Supervision for Cross-lingual Dependency Parsingcited by 8
  3. 2022The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wildcited by 8
  4. 2021Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registerscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 20; 2025: 4; 2026: 1.

45 papers.

Sort:

By area:

  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  3. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  4. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  5. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  6. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  7. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  8. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  9. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  10. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 42
  11. 2023Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instructioncited by 37
  12. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
33 more
  1. 2024'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participantscited by 28
  2. 2023Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparationcited by 16
  3. 2023Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Taskscited by 13
  4. 2023Large Language Models for Travel Behavior Predictioncited by 13
  5. 2024On Limitations of LLM as Annotator for Low Resource Languagescited by 12
  6. 2024Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Educationcited by 11
  7. 2023SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcriptscited by 8
  8. 2023Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweetscited by 8
  9. 2023Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variationcited by 6
  10. 2024Investigating Annotator Bias in Large Language Models for Hate Speech Detectioncited by 6
  11. 2024Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Modelscited by 5
  12. 2025AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbotcited by 5
  13. 2026Gender Dynamics and Homophily in a Social Network of LLM Agentscited by 5
  14. 2023Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Datasetcited by 4
  15. 2024GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Textcited by 4
  16. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  17. 2025Deconstructing Depression Stigma: Integrating AI-driven Data Collection and Analysis with Causal Knowledge Graphscited by 4
  18. 2025Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Taskscited by 4
  19. 2023Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classificationcited by 3
  20. 2024Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Taskscited by 3
  21. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
  22. 2024CoT-Driven Framework for Short Text Classification: Enhancing and Transferring Capabilities from Large to Smaller Modelcited by 2
  23. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
  24. 2024Says Who? Effective Zero-Shot Annotation of Focalizationcited by 2
  25. 2023Analyzing the Impact of Fake News on the Anticipated Outcome of the 2024 Election Ahead of Timecited by 1
  26. 2024Scaling Technology Acceptance Analysis with Large Language Model (LLM) Annotation Systemscited by 1
  27. 2024Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Constructioncited by 1
  28. 2023Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?cited by 0
  29. 2024How to Discern Important Urgent News?cited by 0
  30. 2024Distance Sampling-based Paraphraser Leveraging ChatGPT for Text Data Manipulationcited by 0
  31. 2024Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)cited by 0
  32. 2024Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracycited by 0
  33. 2024Learning to Predict Usage Options of Product Reviews with LLM-Generated Labelscited by 0