# SsciBERT: A Pre-trained Language Model for Social Science Texts

arXiv: https://arxiv.org/abs/2206.04510 · HTML: https://mlsanity.com/paper/2206.04510/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [GREEK-BERT: The Greeks visiting Sesame Street](https://mlsanity.com/paper/2008.12014/) (2020)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting](https://mlsanity.com/paper/2107.00414/) (2021)
- [ExpBERT: Representation Engineering with Natural Language Explanations](https://mlsanity.com/paper/2005.01932/) (2020)
- [ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation](https://mlsanity.com/paper/2101.08382/) (2021)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding](https://mlsanity.com/paper/2105.10912/) (2021)
- [SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP Contributions for a Research Knowledge Graph](https://mlsanity.com/paper/2106.07385/) (2021)
- [Annotating and Extracting Synthesis Process of All-Solid-State Batteries from Scientific Literature](https://mlsanity.com/paper/2002.07339/) (2020)
- [SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts](https://mlsanity.com/paper/2104.08809/) (2021)
- [Natural Language Premise Selection: Finding Supporting Statements for Mathematical Text](https://mlsanity.com/paper/2004.14959/) (2020)
- [Domain-independent Extraction of Scientific Concepts from Research Articles](https://mlsanity.com/paper/2001.03067/) (2020)
- [The STEM-ECR Dataset: Grounding Scientific Entity References in STEM Scholarly Content to Authoritative Encyclopedic and Lexicographic Sources](https://mlsanity.com/paper/2003.01006/) (2020)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [Biomedical Event Extraction with Hierarchical Knowledge Graphs](https://mlsanity.com/paper/2009.09335/) (2020)

## Built on by (most cited first, 12 of 12)

- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN](https://mlsanity.com/paper/2406.12909/) (2024)
- [When Large Language Models Meet Citation: A Survey](https://mlsanity.com/paper/2309.09727/) (2023)
- [Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications](https://mlsanity.com/paper/2209.09062/) (2022)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches](https://mlsanity.com/paper/2503.11376/) (2025)
- [Embedding Models for Supervised Automatic Extraction and Classification of Named Entities in Scientific Acknowledgements](https://mlsanity.com/paper/2307.13377/) (2023)
- [Classifying populist language in American presidential and governor speeches using automatic text analysis](https://mlsanity.com/paper/2408.15213/) (2024)
- [Towards understanding evolution of science through language model series](https://mlsanity.com/paper/2409.09636/) (2024)
- [Enriching Social Science Research via Survey Item Linking](https://mlsanity.com/paper/2412.15831/) (2024)
- [Semantic Embeddings of Chemical Elements for Enhanced Materials Inference and Discovery](https://mlsanity.com/paper/2502.14912/) (2025)
- [SHARE: Social-Humanities AI for Research and Education](https://mlsanity.com/paper/2604.11152/) (2026)

## Abstract (arXiv)

The academic literature of social sciences records human civilization and studies human social problems. With its large-scale growth, the ways to quickly find existing research on relevant issues have become an urgent demand for researchers. Previous studies, such as SciBERT, have shown that pre-training using domain-specific texts can improve the performance of natural language processing tasks. However, the pre-trained language model for social sciences is not available so far. In light of this, the present research proposes a pre-trained model based on the abstracts published in the Social Science Citation Index (SSCI) journals. The models, which are available on GitHub (https://github.com/S-T-Full-Text-Knowledge-Mining/SSCI-BERT), show excellent performance on discipline classification, abstract structure-function recognition, and named entity recognition tasks with the social sciences literature.