mlsanityData to 7 Oct 2026

gaBERT -- an Irish Language Model

arXiv 2107.12930 ↗First posted 27 Jul 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 21 of its 56 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

21 of 56 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
9 more
  1. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  2. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  3. 2019Multilingual is not enough: BERT for Finnishcited by 118
  4. 2019BERTje: A Dutch BERT Modelcited by 109
  5. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  6. 2020ParsBERT: Transformer-based Model for Persian Language Understandingcited by 92
  7. 2020What the [MASK]? Making Sense of Language-Specific BERT Modelscited by 44
  8. 2019Is Multilingual BERT Fluent in Language Generation?cited by 28
  9. 2020WikiBERT models: deep transfer learning for many languagescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 4; 2025: 2; 2026: 2.

12 papers.

  1. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  2. 2024Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translationcited by 9
  3. 2024A Survey of Large Language Models for European Languagescited by 6
  4. 2024UCCIX: Irish-eXcellence Large Language Modelcited by 5
  5. 2023bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmarkcited by 4
  6. 2023Towards spoken dialect identification of Irishcited by 3
  7. 2024Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languagescited by 2
  8. 2025Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Settingcited by 2
  9. 2025Qomhra: A Bilingual Irish and English Large Language Modelcited by 1
  10. 2022Use of Transformer-Based Models for Word-Level Transliteration of the Book of the Dean of Lismorecited by 0
  11. 2026BlasBench: An Open Benchmark for Irish Speech Recognitioncited by 0
  12. 2026MoirfEolas and Cr'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphologycited by 0