# MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media

arXiv: https://arxiv.org/abs/2605.06940 · HTML: https://mlsanity.com/paper/2605.06940/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data](https://mlsanity.com/paper/2402.15343/) (2024)
- [Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-tuned Models for Bangla Sentiment Analysis](https://mlsanity.com/paper/2308.10783/) (2023)
- [BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews](https://mlsanity.com/paper/2305.06595/) (2023)
- [BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis](https://mlsanity.com/paper/2408.08964/) (2024)
- [BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla](https://mlsanity.com/paper/2410.13281/) (2024)
- [SentiGOLD: A Large Bangla Gold Standard Multi-Domain Sentiment Analysis Dataset and its Evaluation](https://mlsanity.com/paper/2306.06147/) (2023)
- [Potrika: Raw and Balanced Newspaper Datasets in the Bangla Language with Eight Topics and Five Attributes](https://mlsanity.com/paper/2210.09389/) (2022)
- [Cost-aware LLM-based Online Dataset Annotation](https://mlsanity.com/paper/2505.15101/) (2025)
- [A Comparative Analysis of Noise Reduction Methods in Sentiment Analysis on Noisy Bangla Texts](https://mlsanity.com/paper/2401.14360/) (2024)
- [Analyzing Emotions in Bangla Social Media Comments Using Machine Learning and LIME](https://mlsanity.com/paper/2506.10154/) (2025)
- [BanglishRev: A Large-Scale Bangla-English and Code-mixed Dataset of Product Reviews in E-Commerce](https://mlsanity.com/paper/2412.13161/) (2024)

## Built on by (most cited first, 1 of 1)

- [SarcasTraps: Effect of Sarcasm on Turkish Hate Speech Detection](https://mlsanity.com/paper/2610.16957/) (2026)

## Abstract (arXiv)

Annotation automation via Large Language Models (LLMs) is the core approach for scaling NLP datasets; however, LLM behavior with respect to closed-set instructions in low-resource languages has not been well studied. We present MultiSoc-4D, a Bengali social media dataset benchmark, which contains 58K+ social media comments from six sources annotated along four dimensions: category, sentiment, hate speech, and sarcasm. By employing a structured pipeline where ChatGPT, Gemini, Claude, and Grok individually annotate separate partitions, while sharing a common validation set of 20%, we diagnose LLM behavior systematically. We discover a prevalent phenomenon called "instruction-induced label collapse", wherein LLMs show a systematic preference towards fallback labels (Other, Neutral, No), leading to high agreement rates but under-detection of minority categories. For example, we find that LLMs failed to detect 79% and 75% of instances with hateful and sarcastic content compared to a human-calibrated reference. Furthermore, we prove that it represents a "label agreement illusion", statistically validated via almost null Fleiss' Kappa (kappa ≈ -0.001) on sarcasm detection. Across 40+ LLMs, we benchmark this annotation bias propagation within the training pipeline, regardless of architectural differences. We release MultiSoc-4D as a diagnostic benchmark for annotation biases in Bengali NLP.