# NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps

arXiv: https://arxiv.org/abs/2404.01651 · HTML: https://mlsanity.com/paper/2404.01651/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 91 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 30 of 34)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://mlsanity.com/paper/1910.03270/) (2019)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech](https://mlsanity.com/paper/2107.08720/) (2021)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [Thou shalt not hate: Countering Online Hate Speech](https://mlsanity.com/paper/1808.04409/) (2018)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [Handling and Presenting Harmful Text in NLP Research](https://mlsanity.com/paper/2204.14256/) (2022)
- [Towards Knowledge-Grounded Counter Narrative Generation for Hate Speech](https://mlsanity.com/paper/2106.11783/) (2021)
- [Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering](https://mlsanity.com/paper/2211.03433/) (2022)
- [Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation](https://mlsanity.com/paper/2303.06433/) (2023)
- [Hate Speech and Counter Speech Detection: Conversational Context Does Matter](https://mlsanity.com/paper/2206.06423/) (2022)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [Defining and Detecting Toxicity on Social Media: Context and Knowledge are Key](https://mlsanity.com/paper/2104.10788/) (2021)
- [From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models](https://mlsanity.com/paper/2305.17174/) (2023)
- [Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language](https://mlsanity.com/paper/2311.00161/) (2023)
- [Countering hate on social media: Large scale classification of hate and counter speech](https://mlsanity.com/paper/2006.01974/) (2020)
- [Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework](https://mlsanity.com/paper/2109.13137/) (2021)
- [Detecting Unintended Social Bias in Toxic Language Datasets](https://mlsanity.com/paper/2210.11762/) (2022)

## Built on by (most cited first, 20 of 20)

- [Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification](https://mlsanity.com/paper/2510.11408/) (2025)
- [HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter](https://mlsanity.com/paper/2411.15462/) (2024)
- [When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits](https://mlsanity.com/paper/2503.03417/) (2025)
- [FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking](https://mlsanity.com/paper/2504.16188/) (2025)
- [On the Role of Speech Data in Reducing Toxicity Detection Bias](https://mlsanity.com/paper/2411.08135/) (2024)
- [Revealing Hidden Mechanisms of Cross-Country Content Moderation with Natural Language Processing](https://mlsanity.com/paper/2503.05280/) (2025)
- [When in Doubt, Cascade: Towards Building Efficient and Capable Guardrails](https://mlsanity.com/paper/2407.06323/) (2024)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [The Enforcement and Feasibility of Hate Speech Moderation](https://mlsanity.com/paper/2604.12289/) (2026)
- [Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure](https://mlsanity.com/paper/2604.25580/) (2026)
- [Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe](https://mlsanity.com/paper/2607.25292/) (2026)
- [Can NLP Tackle Hate Speech in the Real World? Stakeholder-Informed Feedback and Survey on Counterspeech](https://mlsanity.com/paper/2508.04638/) (2025)
- [Are LLMs Good Safety Agents or a Propaganda Engine?](https://mlsanity.com/paper/2511.23174/) (2025)
- [LinGO: A Linguistic Graph Optimization Framework with LLMs for Interpreting Intents of Online Uncivil Discourse](https://mlsanity.com/paper/2602.04693/) (2026)
- [Probing Cultural Signals in Large Language Models through Author Profiling](https://mlsanity.com/paper/2603.16749/) (2026)
- [CHAIRO: Contextual Hierarchical Analogical Induction and Reasoning Optimization for LLMs](https://mlsanity.com/paper/2604.10502/) (2026)
- [Evaluating Large Language Models for Antisemitic Incident Classification](https://mlsanity.com/paper/2607.04890/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation](https://mlsanity.com/paper/2609.03814/) (2026)
- [How Do People Challenge Racial Stereotypes Online? Counter-Story Detection Across Reddit Communities](https://mlsanity.com/paper/2610.04803/) (2026)

## Abstract (arXiv)

The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word. Here we show that computationally modeling this use-mention distinction is crucial for dealing with counterspeech online. Counterspeech that refutes problematic content often mentions harmful language but is not harmful itself (e.g., calling a vaccine dangerous is not the same as expressing disapproval of someone for calling vaccines dangerous). We show that even recent language models fail at distinguishing use from mention, and that this failure propagates to two key downstream tasks: misinformation and hate speech detection, resulting in censorship of counterspeech. We introduce prompting mitigations that teach the use-mention distinction, and show they reduce these errors. Our work highlights the importance of the use-mention distinction for NLP and CSS and offers ways to address it.