mlsanityData to 14 Oct 2026

Evaluation of contextual embeddings on less-resourced languages

arXiv 2107.10614 ↗First posted 22 Jul 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The current dominance of deep neural networks in natural language processing is based on contextual embeddings such as ELMo, BERT, and BERT derivatives. Most existing work focuses on English; in contrast, we present here the first multilingual empirical comparison of two ELMo and several monolingual and multilingual BERT models using 14 tasks in nine languages. In monolingual settings, our analysis shows that monolingual BERT models generally dominate, with a few exceptions such as the dependency parsing task, where they are not competitive with ELMo models trained on large corpora. In cross-lingual settings, BERT models trained on only a few languages mostly do best, closely followed by massively multilingual BERT models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
25 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  2. 2017Word Translation Without Parallel Datacited by 487
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  5. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  6. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  7. 2019Multilingual is not enough: BERT for Finnishcited by 119
  8. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  9. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  10. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  11. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  12. 2019Viable Dependency Parsing as Sequence Labelingcited by 34
  13. 2018Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenationcited by 21
  14. 2021BERTi'c -- The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbiancited by 21
  15. 2020FinEst BERT and CroSloEngual BERT: less is more in multilingual modelscited by 20
  16. 2013DGT-TM: A freely Available Translation Memory in 22 Languagescited by 16
  17. 2020EstBERT: A Pretrained Language-Specific BERT for Estoniancited by 16
  18. 2020Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalizationcited by 15
  19. 2019CoSimLex: A Resource for Evaluating Graded Word Similarity in Contextcited by 13
  20. 2020A Unifying Theory of Transition-based and Sequence Labeling Parsingcited by 12
  21. 2020ELMo and BERT in semantic change detection for Russiancited by 8
  22. 2020MICE: Mining Idioms with Contextual Embeddingscited by 7
  23. 2019High Quality ELMo Embeddings for Seven Less-Resourced Languagescited by 5
  24. 2019Multilingual Culture-Independent Word Analogy Datasetscited by 4
  25. 2021Cross-lingual alignments of ELMo contextual embeddingscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2023Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news mediacited by 14
  2. 2022Sequence to sequence pretraining for a less-resourced Slovenian languagecited by 9
  3. 2024LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classificationcited by 6
  4. 2024Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretrainingcited by 4