# Detecting Hate Speech and Offensive Language on Twitter using Machine Learning: An N-gram and TFIDF based Approach

arXiv: https://arxiv.org/abs/1809.08651 · HTML: https://mlsanity.com/paper/1809.08651/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2018.

## Built on by (most cited first, 10 of 10)

- [A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset](https://mlsanity.com/paper/2308.04037/) (2023)
- [Hate Speech Detection in Roman Urdu](https://mlsanity.com/paper/2108.02830/) (2021)
- [Mitigating Biases in Toxic Language Detection through Invariant Rationalization](https://mlsanity.com/paper/2106.07240/) (2021)
- [Combating Hostility: Covid-19 Fake News and Hostile Post Detection in Social Media](https://mlsanity.com/paper/2101.03291/) (2021)
- [Selecting and combining complementary feature representations and classifiers for hate speech detection](https://mlsanity.com/paper/2201.06721/) (2022)
- [HASOCOne@FIRE-HASOC2020: Using BERT and Multilingual BERT models for Hate Speech Detection](https://mlsanity.com/paper/2101.09007/) (2021)
- [Culture Matters in Toxic Language Detection in Persian](https://mlsanity.com/paper/2506.03458/) (2025)
- [English offensive text detection using CNN based Bi-GRU model](https://mlsanity.com/paper/2409.15652/) (2024)
- [An exploratory experiment on Hindi, Bengali hate-speech detection and transfer learning using neural networks](https://mlsanity.com/paper/2201.01997/) (2022)
- [LinGO: A Linguistic Graph Optimization Framework with LLMs for Interpreting Intents of Online Uncivil Discourse](https://mlsanity.com/paper/2602.04693/) (2026)

## Abstract (arXiv)

Toxic online content has become a major issue in today's world due to an exponential increase in the use of internet by people of different cultures and educational background. Differentiating hate speech and offensive language is a key challenge in automatic detection of toxic text content. In this paper, we propose an approach to automatically classify tweets on Twitter into three classes: hateful, offensive and clean. Using Twitter dataset, we perform experiments considering n-grams as features and passing their term frequency-inverse document frequency (TFIDF) values to multiple machine learning models. We perform comparative analysis of the models considering several values of n in n-grams and TFIDF normalization methods. After tuning the model giving the best results, we achieve 95.6% accuracy upon evaluating it on test data. We also create a module which serves as an intermediate between user and Twitter.