mlsanityData to 14 Oct 2026

Building astroBERT, a language model for Astronomy & Astrophysics

arXiv 2112.00590 ↗First posted 1 Dec 2021cs.CL astro-ph.IMcited by 19

Not on the map. Why?

Abstract (arXiv)

The existing search tools for exploring the NASA Astrophysics Data System (ADS) can be quite rich and empowering (e.g., similar and trending operators), but researchers are not yet allowed to fully leverage semantic search. For example, a query for "results from the Planck mission" should be able to distinguish between all the various meanings of Planck (person, mission, constant, institutions and more) without further clarification from the user. At ADS, we are applying modern machine learning and natural language processing techniques to our dataset of recent astronomy publications to train astroBERT, a deeply contextual language model based on research at Google. Using astroBERT, we aim to enrich the ADS dataset and improve its discoverability, and in particular we are developing our own named entity recognition tool. We present here our preliminary results and lessons learned.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 2 of its 8 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

2 of 8 references.

References · 2

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 6; 2025: 4; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  3. 2023AstroLLaMA: Towards Specialized Foundation Models in Astronomycited by 41
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  5. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  6. 2024Tracing the Development of the Virtual Particle Concept Using Semantic Change Detectioncited by 3
  7. 2024Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of sciencecited by 3
  8. 2025FeynTune: Large Language Models for High-Energy Theorycited by 3
  9. 2022Detecting Entities in the Astrophysics Literature: A Comparison of Word-based and Span-based Entity Recognition Methodscited by 2
  10. 2023NASA Science Mission Directorate Knowledge Graph Discoverycited by 2
  11. 2024Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physicscited by 2
  12. 2024Explaining the Model, Protecting Your Data: Revealing and Mitigating the Data Privacy Risks of Post-Hoc Model Explanations via Membership Inferencecited by 1
7 more
  1. 2022Searching for Carriers of the Diffuse Interstellar Bands Across Disciplines, using Natural Language Processingcited by 0
  2. 2023Identifying Planetary Names in Astronomy Papers: A Multi-Step Approachcited by 0
  3. 2024Psychological Assessments with Large Language Models: A Privacy-Focused and Cost-Effective Approachcited by 0
  4. 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinementcited by 0
  5. 2026AstroConcepts: A Large-Scale Multi-Label Classification Corpus for Astrophysicscited by 0
  6. 2026Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinkingcited by 0
  7. 2026Local Citation Recommendation Without Graph Signalscited by 0