mlsanityData to 14 Oct 2026

Is Multilingual BERT Fluent in Language Generation?

arXiv 1910.03806 ↗First posted 9 Oct 2019cs.CL cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

The multilingual BERT model is trained on 104 languages and meant to serve as a universal language model and tool for encoding sentences. We explore how well the model performs on several languages across several tasks: a diagnostic classification probing the embeddings for a particular syntactic property, a cloze task testing the language modelling ability to fill in gaps in a sentence, and a natural language generation task testing for the ability to produce coherent text fitting a given context. We find that the currently available multilingual BERT model is clearly inferior to the monolingual counterparts, and cannot in many cases serve as a substitute for a well-trained monolingual model. We find that the English and German models perform well at generation, whereas the multilingual model is lacking, in particular, for Nordic languages.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 9; 2023: 2; 2024: 3; 2025: 0; 2026: 1.

28 papers.

Sort:

  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  2. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  3. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  4. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  5. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  6. 2023Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solutioncited by 22
  7. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
  8. 2022Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathicited by 16
  9. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  10. 2021gaBERT -- an Irish Language Modelcited by 12
  11. 2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 12
  12. 2021Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learningcited by 11
16 more
  1. 2021Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Modelscited by 8
  2. 2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6
  3. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  4. 2021John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMscited by 5
  5. 2022State-of-the-art in Open-domain Conversational AI: A Surveycited by 5
  6. 2022Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluationcited by 4
  7. 2022Word-order typology in Multilingual BERT: A case study in subordinate-clause detectioncited by 3
  8. 2021Re-Evaluating GermEval17 Using German Pre-Trained Language Modelscited by 2
  9. 2021Auto-tagging of Short Conversational Sentences using Transformer Methodscited by 2
  10. 2022Understanding BLOOM: An empirical study on diverse NLP taskscited by 2
  11. 2021Vy=akarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languagescited by 1
  12. 2021Transfer Learning for Multi-lingual Tasks -- a Surveycited by 1
  13. 2022AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation of Dialogues in Low-Resource, African Languagescited by 1
  14. 2021Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?cited by 0
  15. 2024Language-Independent Representations Improve Zero-Shot Summarizationcited by 0
  16. 2026YoNER: A New Yor`ub'a Multi-domain Named Entity Recognition Datasetcited by 0