mlsanityData to 7 Oct 2026

MEXMA: Token-level objectives improve sentence representations

arXiv 2409.12737 ↗First posted 19 Sep 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Current pre-trained cross-lingual sentence encoders approaches use sentence-level objectives only. This can lead to loss of information, especially for tokens, which then degrades the sentence representation. We propose MEXMA, a novel approach that integrates both sentence-level and token-level objectives. The sentence representation in one language is used to predict masked tokens in another language, with both the sentence representation and all tokens directly updating the encoder. We show that adding token-level objectives greatly improves the sentence representation quality across several tasks. Our approach outperforms current pre-trained cross-lingual sentence encoders on bi-text mining as well as several downstream tasks. We also analyse the information encoded in our tokens, and how the sentence representation is built from them.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

32 of 44 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 621
20 more
  1. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  2. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  6. 2015Skip-Thought Vectorscited by 321
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  9. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  10. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  11. 2018Spreading vectors for similarity searchcited by 114
  12. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  13. 2020TICO-19: the Translation Initiative for Covid-19cited by 67
  14. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  15. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  16. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  17. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  18. 2013DGT-TM: A freely Available Translation Memory in 22 Languagescited by 16
  19. 2023Dual-Alignment Pre-training for Cross-lingual Sentence Embeddingcited by 9
  20. 2023xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languagescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  2. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  3. 2026Separating Representation from Reconstruction Enables Scalable Text Encoderscited by 1
  4. 2026FLaG: Frequency-Domain Latent-attention Gated Pooling for Token Aggregationcited by 0