mlsanityData to 14 Oct 2026

Incorporating Domain Knowledge into Materials Tokenization

arXiv 2506.11115 ↗First posted 9 Jun 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

While language models are increasingly utilized in materials science, typical models rely on frequency-centric tokenization methods originally developed for natural language processing. However, these methods frequently produce excessive fragmentation and semantic loss, failing to maintain the structural and semantic integrity of material concepts. To address this issue, we propose MATTER, a novel tokenization approach that integrates material knowledge into tokenization. Based on MatDetector trained on our materials knowledge base and a re-ranking method prioritizing material concepts in token merging, MATTER maintains the structural integrity of identified material concepts and prevents fragmentation during tokenization, ensuring their semantic meaning remains intact. The experimental results demonstrate that MATTER outperforms existing tokenization methods, achieving an average performance gain of 4% and 2% in the generation and classification tasks, respectively. These results underscore the importance of domain knowledge for tokenization strategies in scientific text processing. Our code is available at https://github.com/yerimoh/MATTER

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 54 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

22 of 54 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  8. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  9. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  10. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  11. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  12. 2019The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structurescited by 49
10 more
  1. 2023MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modelingcited by 35
  2. 2020The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domaincited by 32
  3. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  4. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 31
  5. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  6. 2023HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Sciencecited by 22
  7. 2021Looking Through Glass: Knowledge Discovery from Materials Science Literature using Natural Language Processingcited by 16
  8. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  9. 2023Biomedical Language Models are Robust to Sub-optimal Tokenizationcited by 9
  10. 2024MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Sciencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026REPAIR: Resolving Long-Tail Confusion in Scientific Retrievers via Fact-Verified Iterative Refinementcited by 1
  2. 2026Exploring LLMs and RAG for Plausible and Explainable Material Prediction of Vehicle Componentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Are LLMs Ready for Real-World Materials Discovery?
  2. 2024Mining experimental data from Materials Science literature with Large Language Models: an evaluation study
  3. 2025Language-Native Materials Processing Design by Lightly Structured Text Database and Reasoning Large Language Model
  4. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
  5. 2025MatterChat: A Multi-Modal LLM for Material Science
  6. 2023MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models
  7. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Science
  8. 2024Foundational Large Language Models for Materials Research