mlsanityData to 14 Oct 2026

HeRo: RoBERTa and Longformer Hebrew Language Models

arXiv 2304.11077 ↗First posted 18 Apr 2023cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we fill in an existing gap in resources available to the Hebrew NLP community by providing it with the largest so far pre-train dataset HeDC4, a state-of-the-art pre-trained language model HeRo for standard length inputs and an efficient transformer LongHeRo for long input sequences. The HeRo model was evaluated on the sentiment analysis, the named entity recognition, and the question answering tasks while the LongHeRo model was evaluated on the document classification task with a dataset composed of long documents. Both HeRo and LongHeRo presented state-of-the-art performance. The dataset and model checkpoints used in this work are publicly available.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

19 of 27 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2019Defending Against Neural Fake Newscited by 559
  12. 2018A Simple Method for Commonsense Reasoningcited by 223
7 more
  1. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  2. 2021AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application Withcited by 28
  3. 2020Neural Modeling for Named Entities and Morphology (NEMO^2)cited by 26
  4. 2022IT5: Text-to-text Pretraining for Italian Language Understanding and Generationcited by 20
  5. 2021HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognitioncited by 14
  6. 2022Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them Allcited by 14
  7. 2021ParaShoot: A Hebrew Question Answering Datasetcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 5; 2026: 2.

12 papers.

  1. 2023DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrewcited by 17
  2. 2025Splintering Nonconcatenative Languages for Better Tokenizationcited by 6
  3. 2023Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrewcited by 5
  4. 2024Leveraging Prompt-Learning for Structured Information Extraction from Crohn's Disease Radiology Reports in a Low-Resource Languagecited by 2
  5. 2023Introducing DictaLM -- A Large Generative Language Model for Modern Hebrewcited by 1
  6. 2025HebID: Detecting Social Identities in Hebrew-language Political Textcited by 1
  7. 2025The Enemy from Within: A Study of Political Delegitimization Discourse in Israeli Political Speechcited by 1
  8. 2025HalleluBERT: Let Every Token That Has Meaning Bear Its Weightcited by 1
  9. 2024Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Languagecited by 0
  10. 2025Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Diseasecited by 0
  11. 2026Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scorescited by 0
  12. 2026The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLPcited by 0