# Is Multilingual BERT Fluent in Language Generation?

arXiv: https://arxiv.org/abs/1910.03806 · HTML: https://mlsanity.com/paper/1910.03806/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 28 of 28)

- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [Towards Best Practices for Training Multilingual Dense Retrieval Models](https://mlsanity.com/paper/2204.02363/) (2022)
- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [Intriguing Properties of Compression on Multilingual Models](https://mlsanity.com/paper/2211.02738/) (2022)
- [gaBERT -- an Irish Language Model](https://mlsanity.com/paper/2107.12930/) (2021)
- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)
- [Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning](https://mlsanity.com/paper/2110.06273/) (2021)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)
- [Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models](https://mlsanity.com/paper/2403.12809/) (2024)
- [John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs](https://mlsanity.com/paper/2106.01060/) (2021)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation](https://mlsanity.com/paper/2207.09076/) (2022)
- [Word-order typology in Multilingual BERT: A case study in subordinate-clause detection](https://mlsanity.com/paper/2205.11987/) (2022)
- [Re-Evaluating GermEval17 Using German Pre-Trained Language Models](https://mlsanity.com/paper/2102.12330/) (2021)
- [Auto-tagging of Short Conversational Sentences using Transformer Methods](https://mlsanity.com/paper/2106.01735/) (2021)
- [Understanding BLOOM: An empirical study on diverse NLP tasks](https://mlsanity.com/paper/2211.14865/) (2022)
- [Vy=akarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages](https://mlsanity.com/paper/2103.00854/) (2021)
- [Transfer Learning for Multi-lingual Tasks -- a Survey](https://mlsanity.com/paper/2110.02052/) (2021)
- [AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation of Dialogues in Low-Resource, African Languages](https://mlsanity.com/paper/2204.08083/) (2022)
- [Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?](https://mlsanity.com/paper/2112.12356/) (2021)
- [Language-Independent Representations Improve Zero-Shot Summarization](https://mlsanity.com/paper/2404.05720/) (2024)
- [YoNER: A New Yor`ub'a Multi-domain Named Entity Recognition Dataset](https://mlsanity.com/paper/2604.05624/) (2026)

## Abstract (arXiv)

The multilingual BERT model is trained on 104 languages and meant to serve as a universal language model and tool for encoding sentences. We explore how well the model performs on several languages across several tasks: a diagnostic classification probing the embeddings for a particular syntactic property, a cloze task testing the language modelling ability to fill in gaps in a sentence, and a natural language generation task testing for the ability to produce coherent text fitting a given context. We find that the currently available multilingual BERT model is clearly inferior to the monolingual counterparts, and cannot in many cases serve as a substitute for a well-trained monolingual model. We find that the English and German models perform well at generation, whereas the multilingual model is lacking, in particular, for Nordic languages.