# Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition

arXiv: https://arxiv.org/abs/2508.08274 · HTML: https://mlsanity.com/paper/2508.08274/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 30 of 48)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Interpretable Neural Predictions with Differentiable Binary Variables](https://mlsanity.com/paper/1905.08160/) (2019)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://mlsanity.com/paper/1910.03270/) (2019)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Thou shalt not hate: Countering Online Hate Speech](https://mlsanity.com/paper/1808.04409/) (2018)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Spread of hate speech in online social media](https://mlsanity.com/paper/1812.01693/) (2018)
- [Probing LLMs for hate speech detection: strengths and vulnerabilities](https://mlsanity.com/paper/2310.12860/) (2023)
- [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://mlsanity.com/paper/2303.04222/) (2023)
- [Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability](https://mlsanity.com/paper/2207.09615/) (2022)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [Detecting East Asian Prejudice on Social Media](https://mlsanity.com/paper/2005.03909/) (2020)

## Built on by (most cited first, 5 of 5)

- [GlassMol: Interpretable Molecular Property Prediction with Concept Bottleneck Models](https://mlsanity.com/paper/2603.01274/) (2026)
- [Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection](https://mlsanity.com/paper/2607.11597/) (2026)
- [J-Miner: Recovering the Decision Logic of Fine-Tuned LLM Classifiers as Compact Rules](https://mlsanity.com/paper/2608.17063/) (2026)
- [The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation](https://mlsanity.com/paper/2608.22956/) (2026)
- [Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages](https://mlsanity.com/paper/2610.18588/) (2026)

## Abstract (arXiv)

The rapid increase in hate speech on social media has exposed an unprecedented impact on society, making automated methods for detecting such content important. Unlike prior black-box models, we propose a novel transparent method for automated hate and counter speech recognition, i.e., "Speech Concept Bottleneck Model" (SCBM), using adjectives as human-interpretable bottleneck concepts. SCBM leverages large language models (LLMs) to map input texts to an abstract adjective-based representation, which is then sent to a light-weight classifier for downstream tasks. Across five benchmark datasets spanning multiple languages and platforms (e.g., Twitter, Reddit, YouTube), SCBM achieves an average macro-F1 score of 0.69 which outperforms the most recently reported results from the literature on four out of five datasets. Aside from high recognition accuracy, SCBM provides a high level of both local and global interpretability. Furthermore, fusing our adjective-based concept representation with transformer embeddings, leads to a 1.8% performance increase on average across all datasets, showing that the proposed representation captures complementary information. Our results demonstrate that adjective-based concept representations can serve as compact, interpretable, and effective encodings for hate and counter speech recognition. With adapted adjectives, our method can also be applied to other NLP tasks.