mlsanityData to 7 Oct 2026

A General-Purpose Multilingual Document Encoder

arXiv 2305.07016 ↗First posted 11 May 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Massively multilingual pretrained transformers (MMTs) have tremendously pushed the state of the art on multilingual NLP and cross-lingual transfer of NLP models in particular. While a large body of work leveraged MMTs to mine parallel data and induce bilingual document embeddings, much less effort has been devoted to training general-purpose (massively) multilingual document encoder that can be used for both supervised and unsupervised document-level tasks. In this work, we pretrain a massively multilingual document encoder as a hierarchical transformer model (HMDE) in which a shallow document transformer contextualizes sentence representations produced by a state-of-the-art pretrained multilingual sentence encoder. We leverage Wikipedia as a readily available source of comparable documents for creating training data, and train HMDE by means of a cross-lingual contrastive objective, further exploiting the category hierarchy of Wikipedia for creation of difficult negatives. We evaluate the effectiveness of HMDE in two arguably most common and prominent cross-lingual document-level tasks: (1) cross-lingual transfer for topical document classification and (2) cross-lingual document retrieval. HMDE is significantly more effective than (i) aggregations of segment-based representations and (ii) multilingual Longformer. Crucially, owing to its massively multilingual lower transformer, HMDE successfully generalizes to languages unseen in document-level pretraining. We publicly release our code and models at https://github.com/ogaloglu/pre-training-multilingual-document-encoders .

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

40 of 53 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
28 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  3. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  4. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  5. 2020Sparse Sinkhorn Attentioncited by 250
  6. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  7. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  8. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  9. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  10. 2019Hierarchical Transformers for Long Document Classificationcited by 103
  11. 2019Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Functioncited by 57
  12. 2019How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptionscited by 48
  13. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  14. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  15. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  16. 2020Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentationcited by 41
  17. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  18. 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
  19. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  20. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31
  21. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  22. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  23. 2019Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learningcited by 16
  24. 2020Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classificationcited by 11
  25. 2019Hierarchical Document Encoder for Parallel Corpus Miningcited by 10
  26. 2017Multilingual Hierarchical Attention Networks for Document Classificationcited by 9
  27. 2020Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover's Distancecited by 7
  28. 2021LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Modelscited by 3

Built on by

Not cited yet.