# Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers

arXiv: https://arxiv.org/abs/2212.00587 · HTML: https://mlsanity.com/paper/2212.00587/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1510.03820/) (2015)
- [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020)
- [Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling](https://mlsanity.com/paper/1611.06639/) (2016)
- [PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data](https://mlsanity.com/paper/2008.09144/) (2020)
- [BiERU: Bidirectional Emotional Recurrent Unit for Conversational Sentiment Analysis](https://mlsanity.com/paper/2006.00492/) (2020)
- [Words are not Equal: Graded Weighting Model for building Composite Document Vectors](https://mlsanity.com/paper/1512.03549/) (2015)
- [Sentiment Analysis on Brazilian Portuguese User Reviews](https://mlsanity.com/paper/2112.05459/) (2021)
- [BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives](https://mlsanity.com/paper/2201.03382/) (2022)

## Built on by (most cited first, 2 of 2)

- [CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records](https://mlsanity.com/paper/2506.15118/) (2025)
- [Explicit Grammar Semantic Feature Fusion for Robust Text Classification](https://mlsanity.com/paper/2602.20749/) (2026)

## Abstract (arXiv)

Text classification is a natural language processing (NLP) task relevant to many commercial applications, like e-commerce and customer service. Naturally, classifying such excerpts accurately often represents a challenge, due to intrinsic language aspects, like irony and nuance. To accomplish this task, one must provide a robust numerical representation for documents, a process known as embedding. Embedding represents a key NLP field nowadays, having faced a significant advance in the last decade, especially after the introduction of the word-to-vector concept and the popularization of Deep Learning models for solving NLP tasks, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer-based Language Models (TLMs). Despite the impressive achievements in this field, the literature coverage regarding generating embeddings for Brazilian Portuguese texts is scarce, especially when considering commercial user reviews. Therefore, this work aims to provide a comprehensive experimental study of embedding approaches targeting a binary sentiment classification of user reviews in Brazilian Portuguese. This study includes from classical (Bag-of-Words) to state-of-the-art (Transformer-based) NLP models. The methods are evaluated with five open-source databases with pre-defined data partitions made available in an open digital repository to encourage reproducibility. The Fine-tuned TLMs achieved the best results for all cases, being followed by the Feature-based TLM, LSTM, and CNN, with alternate ranks, depending on the database under analysis.