# An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification

arXiv: https://arxiv.org/abs/2412.17361 · HTML: https://mlsanity.com/paper/2412.17361/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2024.

## Built on by (most cited first, 1 of 1)

- [I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System](https://mlsanity.com/paper/2606.11875/) (2026)

## Abstract (arXiv)

This study investigates the performance of three popular tokenization tools: MeCab, Sudachi, and SentencePiece, when applied as a preprocessing step for sentiment-based text classification of Japanese texts. Using Term Frequency-Inverse Document Frequency (TF-IDF) vectorization, we evaluate two traditional machine learning classifiers: Multinomial Naive Bayes and Logistic Regression. The results reveal that Sudachi produces tokens closely aligned with dictionary definitions, while MeCab and SentencePiece demonstrate faster processing speeds. The combination of SentencePiece, TF-IDF, and Logistic Regression outperforms the other alternatives in terms of classification performance.