# HateCheckHIn: Evaluating Hindi Hate Speech Detection Models

arXiv: https://arxiv.org/abs/2205.00328 · HTML: https://mlsanity.com/paper/2205.00328/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 30 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Offensive Language Identification in Greek](https://mlsanity.com/paper/2003.07459/) (2020)
- [Offensive Language and Hate Speech Detection for Danish](https://mlsanity.com/paper/1908.04531/) (2019)
- [Tackling Online Abuse: A Survey of Automated Abuse Detection Methods](https://mlsanity.com/paper/1908.06024/) (2019)
- [Hostility Detection Dataset in Hindi](https://mlsanity.com/paper/2011.03588/) (2020)
- [The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions](https://mlsanity.com/paper/2004.13606/) (2020)
- [Abusive and Threatening Language Detection in Urdu using Boosting based and BERT based models: A Comparative Approach](https://mlsanity.com/paper/2111.14830/) (2021)

## Built on by (most cited first, 15 of 15)

- [Airavata: Introducing Hindi Instruction-tuned LLM](https://mlsanity.com/paper/2401.15006/) (2024)
- [BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla](https://mlsanity.com/paper/2410.13281/) (2024)
- [A Graph-Based Context-Aware Model to Understand Online Conversations](https://mlsanity.com/paper/2211.09207/) (2022)
- [Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection](https://mlsanity.com/paper/2305.13276/) (2023)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [Improving code-mixed hate detection by native sample mixing: A case study for Hindi-English code-mixed scenario](https://mlsanity.com/paper/2405.20755/) (2024)
- [Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection](https://mlsanity.com/paper/2412.12761/) (2024)
- [Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models?](https://mlsanity.com/paper/2506.12744/) (2025)
- [Multilingual Hate Speech Detection and Counterspeech Generation: A Comprehensive Survey and Practical Guide](https://mlsanity.com/paper/2603.19279/) (2026)
- [Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure](https://mlsanity.com/paper/2604.25580/) (2026)
- [Think Outside the Data: Colonial Biases and Systemic Issues in Automated Moderation Pipelines for Low-Resource Languages](https://mlsanity.com/paper/2501.13836/) (2025)
- [Leveraging the Potential of Prompt Engineering for Hate Speech Detection in Low-Resource Languages](https://mlsanity.com/paper/2506.23930/) (2025)
- [TeamHerald@CHIPSAL 2026: Hate Speech Detection and Sentiment Analysis of Nepali Memes using Transformer-based Architectures and Ensemble Learning](https://mlsanity.com/paper/2606.08770/) (2026)
- [Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection](https://mlsanity.com/paper/2609.22261/) (2026)

## Abstract (arXiv)

Due to the sheer volume of online hate, the AI and NLP communities have started building models to detect such hateful content. Recently, multilingual hate is a major emerging challenge for automated detection where code-mixing or more than one language have been used for conversation in social media. Typically, hate speech detection models are evaluated by measuring their performance on the held-out test data using metrics such as accuracy and F1-score. While these metrics are useful, it becomes difficult to identify using them where the model is failing, and how to resolve it. To enable more targeted diagnostic insights of such multilingual hate speech models, we introduce a set of functionalities for the purpose of evaluation. We have been inspired to design this kind of functionalities based on real-world conversation on social media. Considering Hindi as a base language, we craft test cases for each functionality. We name our evaluation dataset HateCheckHIn. To illustrate the utility of these functionalities , we test state-of-the-art transformer based m-BERT model and the Perspective API.