# K-12BERT: BERT for K-12 education

arXiv: https://arxiv.org/abs/2205.12335 · HTML: https://mlsanity.com/paper/2205.12335/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model](https://mlsanity.com/paper/1906.02124/) (2019)
- [Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation](https://mlsanity.com/paper/2109.01048/) (2021)
- [TagRec: Automated Tagging of Questions with Hierarchical Learning Taxonomy](https://mlsanity.com/paper/2107.10649/) (2021)

## Built on by (most cited first, 1 of 1)

- [Cross Encoding as Augmentation: Towards Effective Educational Text Classification](https://mlsanity.com/paper/2305.18977/) (2023)

## Abstract (arXiv)

Online education platforms are powered by various NLP pipelines, which utilize models like BERT to aid in content curation. Since the inception of the pre-trained language models like BERT, there have also been many efforts toward adapting these pre-trained models to specific domains. However, there has not been a model specifically adapted for the education domain (particularly K-12) across subjects to the best of our knowledge. In this work, we propose to train a language model on a corpus of data curated by us across multiple subjects from various sources for K-12 education. We also evaluate our model, K12-BERT, on downstream tasks like hierarchical taxonomy tagging.