mlsanityData to 7 Oct 2026

Evaluating Embedding Frameworks for Scientific Domain

arXiv 2510.06244 ↗First posted 3 Oct 2025cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Finding an optimal word representation algorithm is particularly important in terms of domain specific data, as the same word can have different meanings and hence, different representations depending on the domain and context. While Generative AI and transformer architecture does a great job at generating contextualized embeddings for any given work, they are quite time and compute extensive, especially if we were to pre-train such a model from scratch. In this work, we focus on the scientific domain and finding the optimal word representation algorithm along with the tokenization method that could be used to represent words in the scientific domain. The goal of this research is two fold: 1) finding the optimal word representation and tokenization methods that can be used in downstream scientific domain NLP tasks, and 2) building a comprehensive evaluation suite that could be used to evaluate various word representation and tokenization algorithms (even as new ones are introduced) in the scientific domain. To this end, we build an evaluation suite consisting of several downstream tasks and relevant datasets for each task. Furthermore, we use the constructed evaluation suite to test various word representation and tokenization algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

16 of 33 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  12. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
4 more
  1. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  2. 2019Few-Shot Representation Learning for Out-Of-Vocabulary Wordscited by 30
  3. 2019Long-length Legal Document Classificationcited by 10
  4. 2020Deep learning models for representing out-of-vocabulary wordscited by 4

Built on by

Not cited yet.