mlsanityData to 7 Oct 2026

Probing Multilingual Language Models for Discourse

arXiv 2106.04832 ↗First posted 9 Jun 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Pre-trained multilingual language models have become an important building block in multilingual natural language processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge across languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

25 of 46 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2019How multilingual is Multilingual BERT?cited by 642
13 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  2. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  3. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  4. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  5. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  6. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  7. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  8. 2019Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claimscited by 73
  9. 2020X-Stance: A Multilingual Multi-Target Dataset for Stance Detectioncited by 42
  10. 2019Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representationscited by 21
  11. 2017Cross-lingual RST Discourse Parsingcited by 19
  12. 2020Shallow Discourse Annotation for Chinese TED Talkscited by 13
  13. 2019Zero-shot transfer for implicit discourse relation classificationcited by 5

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 2; 2024: 3; 2025: 1.

12 papers.

  1. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  2. 2021Pragmatic competence of pre-trained language models through the lens of discourse connectivescited by 16
  3. 2024eRST: A Signaled Graph Theory of Discourse Relations and Organizationcited by 13
  4. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8
  5. 2022SocioProbe: What, When, and Where Language Models Learn about Sociodemographicscited by 7
  6. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  7. 2024Bilingual Rhetorical Structure Parsing with Large Parallel Annotationscited by 5
  8. 2025Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Setcited by 4
  9. 2022Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluationcited by 3
  10. 2022"No, they did not": Dialogue response dynamics in pre-trained language modelscited by 1
  11. 2023How Does Pretraining Improve Discourse-Aware Translation?cited by 1
  12. 2023A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classificationcited by 0