mlsanityData to 7 Oct 2026

PhysBERT: A Text Embedding Model for Physics Scientific Literature

arXiv 2408.09574 ↗First posted 18 Aug 2024physics.comp-ph cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The specialized language and complex concepts in physics pose significant challenges for information extraction through Natural Language Processing (NLP). Central to effective NLP applications is the text embedding model, which converts text into dense vector representations for efficient information retrieval and semantic analysis. In this work, we introduce PhysBERT, the first physics-specific text embedding model. Pre-trained on a curated corpus of 1.2 million arXiv physics papers and fine-tuned with supervised data, PhysBERT outperforms leading general-purpose models on physics-specific tasks including the effectiveness in fine-tuning for specific physics subdomains.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

29 of 56 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
17 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  4. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  7. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  8. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  9. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  10. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
  11. 2021MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extractioncited by 82
  12. 2023Information Retrieval: Recent Advances and Beyondcited by 49
  13. 2023Chain-of-Factors Paper-Reviewer Matchingcited by 11
  14. 2023Structure to Property: Chemical Element Embeddings and a Deep Learning Approach for Accurate Prediction of Chemical Propertiescited by 6
  15. 2024Towards Unlocking Insights from Logbooks Using AIcited by 5
  16. 2023PACuna: Automated Fine-Tuning of Language Models for Particle Acceleratorscited by 3
  17. 2023Textual Analysis of ICALEPCS and IPAC Conference Proceedings: Revealing Research Trends, Topics, and Collaborations for Future Insights and Advanced Searchcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 5.

8 papers.

  1. 2025ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddingscited by 4
  2. 2024Tracing the Development of the Virtual Particle Concept Using Semantic Change Detectioncited by 3
  3. 2024Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of sciencecited by 3
  4. 2025T-VEC: A Telecom-Specific Vectorization Model with Enhanced Semantic Understanding via Deep Triplet Loss Fine-Tuningcited by 3
  5. 2024Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physicscited by 2
  6. 2025FeynTune: Large Language Models for High-Energy Theorycited by 2
  7. 2025ComProScanner: A multi-agent based framework for composition-property structured data extraction from scientific literaturecited by 1
  8. 2025PINE: Pipeline for Important Node Exploration in Attributed Networkscited by 1