# Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings?

arXiv: https://arxiv.org/abs/2304.14796 · HTML: https://mlsanity.com/paper/2304.14796/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation](https://mlsanity.com/paper/1607.05368/) (2016)
- [Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages](https://mlsanity.com/paper/2205.12654/) (2022)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [Efficient Classification of Long Documents Using Transformers](https://mlsanity.com/paper/2203.11258/) (2022)
- [Exploiting Sentence Order in Document Alignment](https://mlsanity.com/paper/2004.14523/) (2020)
- [Transformer based Multilingual document Embedding model](https://mlsanity.com/paper/2008.08567/) (2020)

## Built on by (most cited first, 1 of 1)

- [Towards an automatic method for generating topical vocabulary test forms for specific reading passages](https://mlsanity.com/paper/2505.18762/) (2025)

## Abstract (arXiv)

Dense vector representations for textual data are crucial in modern NLP. Word embeddings and sentence embeddings estimated from raw texts are key in achieving state-of-the-art results in various tasks requiring semantic understanding. However, obtaining embeddings at the document level is challenging due to computational requirements and lack of appropriate data. Instead, most approaches fall back on computing document embeddings based on sentence representations. Although there exist architectures and models to encode documents fully, they are in general limited to English and few other high-resourced languages. In this work, we provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We compare input token number truncation, sentence averaging as well as some simple windowing and in some cases new augmented and learnable approaches, on 3 multi- and cross-lingual tasks in 8 languages belonging to 3 different language families. Our task-based extrinsic evaluations show that, independently of the language, a clever combination of sentence embeddings is usually better than encoding the full document as a single unit, even when this is possible. We demonstrate that while a simple sentence average results in a strong baseline for classification tasks, more complex combinations are necessary for semantic tasks.