# On Limitations of LLM as Annotator for Low Resource Languages

arXiv: https://arxiv.org/abs/2411.17637 · HTML: https://mlsanity.com/paper/2411.17637/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 15 of 15)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset](https://mlsanity.com/paper/2103.11408/) (2021)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation](https://mlsanity.com/paper/2405.01299/) (2024)
- [L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models](https://mlsanity.com/paper/2203.13778/) (2022)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [L3Cube-MahaSent-MD: A Multi-domain Marathi Sentiment Analysis Dataset and Transformer Models](https://mlsanity.com/paper/2306.13888/) (2023)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)

## Built on by (most cited first, 12 of 12)

- [From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling](https://mlsanity.com/paper/2506.16393/) (2025)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs](https://mlsanity.com/paper/2508.19089/) (2025)
- [Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study](https://mlsanity.com/paper/2507.14304/) (2025)
- [Agentic AI for autonomous anomaly management in complex systems](https://mlsanity.com/paper/2507.15676/) (2025)
- [Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining](https://mlsanity.com/paper/2512.12770/) (2025)
- [Analysis of LLM as a grammatical feature tagger for African American English](https://mlsanity.com/paper/2502.06004/) (2025)
- [L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models](https://mlsanity.com/paper/2506.00863/) (2025)
- [MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models](https://mlsanity.com/paper/2508.17444/) (2025)
- [Are LLMs Ready to Replace Bangla Annotators?](https://mlsanity.com/paper/2602.16241/) (2026)
- [Structure-Preserving Document Translation via Multi-Stage LLM Pipeline: A Case Study in Marathi](https://mlsanity.com/paper/2606.28796/) (2026)
- [Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA](https://mlsanity.com/paper/2607.08731/) (2026)

## Abstract (arXiv)

Low-resource languages face significant challenges due to the lack of sufficient linguistic data, resources, and tools for tasks such as supervised learning, annotation, and classification. This shortage hinders the development of accurate models and datasets, making it difficult to perform critical NLP tasks like sentiment analysis or hate speech detection. To bridge this gap, Large Language Models (LLMs) present an opportunity for potential annotators, capable of generating datasets and resources for these underrepresented languages. In this paper, we focus on Marathi, a low-resource language, and evaluate the performance of both closed-source and open-source LLMs as annotators, while also comparing these results with fine-tuned BERT models. We assess models such as GPT-4o and Gemini 1.0 Pro, Gemma 2 (2B and 9B), and Llama 3.1 (8B and 405B) on classification tasks including sentiment analysis, news classification, and hate speech detection. Our findings reveal that while LLMs excel in annotation tasks for high-resource languages like English, they still fall short when applied to Marathi. Even advanced models like GPT-4o and Llama 3.1 405B underperform compared to fine-tuned BERT-based baselines, with GPT-4o and Llama 3.1 405B trailing fine-tuned BERT by accuracy margins of 10.2% and 14.1%, respectively. This highlights the limitations of LLMs as annotators for low-resource languages.