mlsanityData to 14 Oct 2026

Bertinho: Galician BERT Representations

arXiv 2103.13799 ↗First posted 25 Mar 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper presents a monolingual BERT model for Galician. We follow the recent trend that shows that it is feasible to build robust monolingual BERT models even for relatively low-resource languages, while performing better than the well-known official multilingual BERT (mBERT). More particularly, we release two monolingual Galician BERT models, built using 6 and 12 transformer layers, respectively; trained with limited resources (~45 million tokens on a single GPU of 24GB). We then provide an exhaustive evaluation on a number of tasks such as POS-tagging, dependency parsing and named entity recognition. For this purpose, all these tasks are cast in a pure sequence labeling setup in order to run BERT without the need to include any additional layers on top of it (we only use an output classification layer to map the contextualized representations into the predicted label). The experiments show that our models, especially the 12-layer one, outperform the results of mBERT in most tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

32 of 58 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  12. 2019How multilingual is Multilingual BERT?cited by 645
20 more
  1. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  3. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  4. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  5. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  6. 2015Semi-supervised Sequence Learningcited by 198
  7. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  8. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  9. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  10. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  11. 2019Multilingual is not enough: BERT for Finnishcited by 119
  12. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  13. 2019Portuguese Named Entity Recognition using BERT-CRFcited by 73
  14. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 38
  15. 2019Viable Dependency Parsing as Sequence Labelingcited by 34
  16. 2020Give your Text Representation Models some Love: the Case for Basquecited by 28
  17. 2020Parsing as Pretrainingcited by 19
  18. 2020KR-BERT: A Small-Scale Korean-Specific Language Modelcited by 17
  19. 2017Contextualized Word Representations for Reading Comprehensioncited by 5
  20. 2018Transition-based Parsing with Lighter Feed-Forward Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2; 2024: 1.

7 papers.

  1. 2022A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Modelscited by 13
  2. 2022drsphelps at SemEval-2022 Task 2: Learning idiom representations using BERTRAMcited by 5
  3. 2023Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representationscited by 4
  4. 2022Sample Efficient Approaches for Idiomaticity Detectioncited by 3
  5. 2022A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training timecited by 3
  6. 2024Open Generative Large Language Models for Galiciancited by 3
  7. 2023Unsupervised Paraphrasing of Multiword Expressionscited by 0