mlsanityData to 7 Oct 2026

Are the Multilingual Models Better? Improving Czech Sentiment with Transformers

arXiv 2108.10640 ↗First posted 24 Aug 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we aim at improving Czech sentiment with transformer-based models and their multilingual versions. More concretely, we study the task of polarity detection for the Czech language on three sentiment polarity datasets. We fine-tune and perform experiments with five multilingual and three monolingual models. We compare the monolingual and multilingual models' performance, including comparison with the older approach based on recurrent neural networks. Furthermore, we test the multilingual models and their ability to transfer knowledge from English to Czech (and vice versa) with zero-shot cross-lingual classification. Our experiments show that the huge multilingual models can overcome the performance of the monolingual models. They are also able to detect polarity in another language without any training data, with performance not worse than 4.4 % compared to state-of-the-art monolingual trained models. Moreover, we achieved new state-of-the-art results on all three datasets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 24 Aug 2021.

Builds on

34 of 64 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
22 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  5. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  6. 2019CamemBERT: a Tasty French Language Modelcited by 368
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  9. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  10. 2019Multilingual is not enough: BERT for Finnishcited by 118
  11. 2019BERTje: A Dutch BERT Modelcited by 109
  12. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  13. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  14. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  15. 2015Reasoning about Entailment with Neural Attentioncited by 68
  16. 2020The birth of Romanian BERTcited by 44
  17. 2015Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problemscited by 23
  18. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  19. 2018Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translationcited by 19
  20. 2017BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMscited by 16
  21. 2016Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddingscited by 11
  22. 2018Linear Transformations for Cross-lingual Semantic Textual Similaritycited by 3

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 0; 2025: 2.

7 papers.

  1. 2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 11
  2. 2022Czech Dataset for Cross-lingual Subjectivity Classificationcited by 5
  3. 2023Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Modelcited by 5
  4. 2022Linear Transformations for Cross-lingual Sentiment Analysiscited by 2
  5. 2025Prompt-Based Approach for Czech Sentiment Analysiscited by 2
  6. 2025Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challengescited by 1
  7. 2023UBC-DLNLP at SemEval-2023 Task 12: Impact of Transfer Learning on African Sentiment Analysiscited by 0