# Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA

arXiv: https://arxiv.org/abs/2104.02041 · HTML: https://mlsanity.com/paper/2104.02041/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)

## Built on by (most cited first, 10 of 10)

- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [A Review on Text-Based Emotion Detection -- Techniques, Applications, Datasets, and Future Directions](https://mlsanity.com/paper/2205.03235/) (2022)
- [RaFoLa: A Rationale-Annotated Corpus for Detecting Indicators of Forced Labour](https://mlsanity.com/paper/2205.02684/) (2022)
- [A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2204.04980/) (2022)
- [Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting](https://mlsanity.com/paper/2402.14184/) (2024)
- [NepEMO: A Multi-Label Emotion and Sentiment Analysis on Nepali Reddit with Linguistic Insights and Temporal Trends](https://mlsanity.com/paper/2512.22823/) (2025)
- [Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization](https://mlsanity.com/paper/2602.09121/) (2026)
- [TamilEmo: Finegrained Emotion Detection Dataset for Tamil](https://mlsanity.com/paper/2202.04725/) (2022)
- [CEFER: A Four Facets Framework based on Context and Emotion embedded features for Implicit and Explicit Emotion Recognition](https://mlsanity.com/paper/2209.13999/) (2022)
- [Improved Text Emotion Prediction Using Combined Valence and Arousal Ordinal Classification](https://mlsanity.com/paper/2404.01805/) (2024)

## Abstract (arXiv)

This paper investigates how Natural Language Understanding (NLU) could be applied in Emotion Recognition, a specific task in affective computing. We finetuned different transformers language models (BERT, DistilBERT, RoBERTa, XLNet, and ELECTRA) using a fine-grained emotion dataset and evaluating them in terms of performance (f1-score) and time to complete.