# HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

arXiv: https://arxiv.org/abs/2102.01909 · HTML: https://mlsanity.com/paper/2102.01909/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 82 references that are in the map.
- First posted on arXiv 3 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Deep Learning for Sentiment Analysis : A Survey](https://mlsanity.com/paper/1801.07883/) (2018)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [Comparative Study of CNN and RNN for Natural Language Processing](https://mlsanity.com/paper/1702.01923/) (2017)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?](https://mlsanity.com/paper/2005.01330/) (2020)
- [Neural Modeling for Named Entities and Morphology (NEMO^2)](https://mlsanity.com/paper/2007.15620/) (2020)
- [Deep learning for affective computing: text-based emotion recognition in decision support](https://mlsanity.com/paper/1803.06397/) (2018)
- [What do character-level models learn about morphology? The case of dependency parsing](https://mlsanity.com/paper/1808.09180/) (2018)
- [Alternative structures for character-level RNNs](https://mlsanity.com/paper/1511.06303/) (2015)

## Built on by (most cited first, 14 of 14)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With](https://mlsanity.com/paper/2104.04052/) (2021)
- [Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://mlsanity.com/paper/2211.15199/) (2022)
- [Breaking Character: Are Subwords Good Enough for MRLs After All?](https://mlsanity.com/paper/2204.04748/) (2022)
- [ParaShoot: A Hebrew Question Answering Dataset](https://mlsanity.com/paper/2109.11314/) (2021)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://mlsanity.com/paper/2304.11077/) (2023)
- [Offensive Hebrew Corpus and Detection using BERT](https://mlsanity.com/paper/2309.02724/) (2023)
- [Detecting Suicide Risk in Online Counseling Services: A Study in a Low-Resource Language](https://mlsanity.com/paper/2209.04830/) (2022)
- [Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language](https://mlsanity.com/paper/2208.01875/) (2022)
- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://mlsanity.com/paper/2212.09682/) (2022)
- [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://mlsanity.com/paper/2405.07099/) (2024)
- [Style transfer and classification in hebrew news items](https://mlsanity.com/paper/2212.03019/) (2022)
- [A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry](https://mlsanity.com/paper/2402.17371/) (2024)

## Abstract (arXiv)

This paper introduces HeBERT and HebEMO. HeBERT is a Transformer-based model for modern Hebrew text, which relies on a BERT (Bidirectional Encoder Representations for Transformers) architecture. BERT has been shown to outperform alternative architectures in sentiment analysis, and is suggested to be particularly appropriate for MRLs. Analyzing multiple BERT specifications, we find that while model complexity correlates with high performance on language tasks that aim to understand terms in a sentence, a more-parsimonious model better captures the sentiment of entire sentence. Either way, out BERT-based language model outperforms all existing Hebrew alternatives on all common language tasks. HebEMO is a tool that uses HeBERT to detect polarity and extract emotions from Hebrew UGC. HebEMO is trained on a unique Covid-19-related UGC dataset that we collected and annotated for this study. Data collection and annotation followed an active learning procedure that aimed to maximize predictability. We show that HebEMO yields a high F1-score of 0.96 for polarity classification. Emotion detection reaches F1-scores of 0.78-0.97 for various target emotions, with the exception of surprise, which the model failed to capture (F1 = 0.41). These results are better than the best-reported performance, even among English-language models of emotion detection.