mlsanityData to 7 Oct 2026

OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement

arXiv 2309.10539 ↗First posted 19 Sep 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We develop and evaluate multilingual scientific documents similarity measurement models in this work. Such models can be used to find related works in different languages, which can help multilingual researchers find and explore papers more efficiently. We propose the first multilingual scientific documents dataset, Open-access Multilingual Scientific Documents (OpenMSD), which has 74M papers in 103 languages and 778M citation pairs. With OpenMSD, we pretrain science-specialized language models, and explore different strategies to derive "related" paper pairs to fine-tune the models, including using a mixture of citation, co-citation, and bibliographic-coupling pairs. To further improve the models' performance for non-English papers, we explore the use of generative language models to enrich the non-English papers with English summaries. This allows us to leverage the models' English capabilities to create better representations for non-English papers. Our best model significantly outperforms strong baselines by 7-16% (in mean average precision).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 85 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

46 of 85 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
34 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  14. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  15. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  16. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  17. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  18. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  19. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  20. 2019PyTorch-BigGraph: A Large-scale Graph Embedding Systemcited by 118
  21. 2023The Semantic Scholar Open Data Platformcited by 109
  22. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  23. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  24. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  25. 2019NCLS: Neural Cross-Lingual Summarizationcited by 64
  26. 2018End-to-End Retrieval in Continuous Spacecited by 46
  27. 2022A Survey on Cross-Lingual Summarizationcited by 38
  28. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  29. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  30. 2019OpenCitations, an infrastructure organization for open scholarshipcited by 16
  31. 2020InstantEmbedding: Efficient Local Node Representationscited by 9
  32. 2019Efficient Inner Product Approximation in Hybrid Spacescited by 6
  33. 2022MACRONYM: A Large-Scale Dataset for Multilingual and Multi-Domain Acronym Extractioncited by 4
  34. 2019Non-English language publications in Citation Indexes -- quantity and qualitycited by 2

Built on by

Not cited yet.