mlsanityData to 13 Oct 2026

Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models

arXiv 2406.13695 ↗First posted 19 Jun 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper addresses the deduplication of multilingual textual data using advanced NLP tools. We compare a two-step method involving translation to English followed by embedding with mpnet, and a multilingual embedding model (distiluse). The two-step approach achieved a higher F1 score (82% vs. 60%), particularly with less widely used languages, which can be increased up to 89% by leveraging expert rules based on domain knowledge. We also highlight limitations related to token length constraints and computational efficiency. Our methodology suggests improvements for future multilingual deduplication tasks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

19 of 29 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  6. 2017Billion-scale similarity search with GPUscited by 2,490
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 976
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 710
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  10. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  11. 2014word2vec Parameter Learning Explainedcited by 53
  12. 2020A Survey on Deep Hashing Methodscited by 47
7 more
  1. 2020Comparing BERT against traditional machine learning text classificationcited by 46
  2. 2021GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrievalcited by 39
  3. 2020A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Textscited by 21
  4. 2022CultureBERT: Measuring Corporate Culture With Transformer-Based Language Modelscited by 3
  5. 2023Ahead of the Text: Leveraging Entity Preposition for Financial Relation Extractioncited by 2
  6. 2021Projective Clustering Product Quantizationcited by 1
  7. 2022StonkBERT: Can Language Models Predict Medium-Run Stock Price Movements?cited by 1

Built on by

Not cited yet.