# Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classification

arXiv: https://arxiv.org/abs/2206.02206 · HTML: https://mlsanity.com/paper/2206.02206/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 5 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Multi-Head Attention: Collaborate Instead of Concatenate](https://mlsanity.com/paper/2006.16362/) (2020)
- [k-Nearest Neighbour Classifiers: 2nd Edition (with Python examples)](https://mlsanity.com/paper/2004.04523/) (2020)
- [Word Embeddings: A Survey](https://mlsanity.com/paper/1901.09069/) (2019)
- [A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models](https://mlsanity.com/paper/2010.15036/) (2020)
- [Similarity encoding for learning with dirty categorical variables](https://mlsanity.com/paper/1806.00979/) (2018)
- [Neural Machine Translation: A Review of Methods, Resources, and Tools](https://mlsanity.com/paper/2012.15515/) (2020)
- [Max-Pooling Dropout for Regularization of Convolutional Neural Networks](https://mlsanity.com/paper/1512.01400/) (2015)
- [Recent Trends in Named Entity Recognition (NER)](https://mlsanity.com/paper/2101.11420/) (2021)
- [Succinct Differentiation of Disparate Boosting Ensemble Learning Methods for Prognostication of Polycystic Ovary Syndrome Diagnosis](https://mlsanity.com/paper/2201.00418/) (2022)
- [Discriminant Analysis in Contrasting Dimensions for Polycystic Ovary Syndrome Prognostication](https://mlsanity.com/paper/2201.03029/) (2022)
- [Binary Classification for High Dimensional Data using Supervised Non-Parametric Ensemble Method](https://mlsanity.com/paper/2202.07779/) (2022)
- [Selective Feature Connection Mechanism: Concatenating Multi-layer CNN Features with a Feature Selector](https://mlsanity.com/paper/1811.06295/) (2018)
- [Combining Varied Learners for Binary Classification using Stacked Generalization](https://mlsanity.com/paper/2202.08910/) (2022)
- [Effects of Parametric and Non-Parametric Methods on High Dimensional Sparse Matrix Representations](https://mlsanity.com/paper/2202.02894/) (2022)
- [Res-CNN-BiLSTM Network for overcoming Mental Health Disturbances caused due to Cyberbullying through Social Media](https://mlsanity.com/paper/2204.09738/) (2022)

## Abstract (arXiv)

The task of text classification using Bidirectional based LSTM architectures is computationally expensive and time consuming to train. For this, transformers were discovered which effectively give good performance as compared to the traditional deep learning architectures. In this paper we present a performance based comparison between simple transformer based network and Res-CNN-BiLSTM based network for cyberbullying text classification problem. The results obtained show that transformer we trained with 0.65 million parameters has significantly being able to beat the performance of Res-CNN-BiLSTM with 48.82 million parameters for faster training speeds and more generalized metrics. The paper also compares the 1-dimensional character level embedding network and 100-dimensional glove embedding network with transformer.