# Enriched BERT Embeddings for Scholarly Publication Classification

arXiv: https://arxiv.org/abs/2405.04136 · HTML: https://mlsanity.com/paper/2405.04136/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering](https://mlsanity.com/paper/1811.11934/) (2018)
- [Enriching BERT with Knowledge Graph Embeddings for Document Classification](https://mlsanity.com/paper/1909.08402/) (2019)
- [Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?](https://mlsanity.com/paper/2101.08114/) (2021)

## Built on by (most cited first, 1 of 1)

- [SDEC: Semantic Deep Embedded Clustering](https://mlsanity.com/paper/2508.15823/) (2025)

## Abstract (arXiv)

With the rapid expansion of academic literature and the proliferation of preprints, researchers face growing challenges in manually organizing and labeling large volumes of articles. The NSLP 2024 FoRC Shared Task I addresses this challenge organized as a competition. The goal is to develop a classifier capable of predicting one of 123 predefined classes from the Open Research Knowledge Graph (ORKG) taxonomy of research fields for a given article.This paper presents our results. Initially, we enrich the dataset (containing English scholarly articles sourced from ORKG and arXiv), then leverage different pre-trained language Models (PLMs), specifically BERT, and explore their efficacy in transfer learning for this downstream task. Our experiments encompass feature-based and fine-tuned transfer learning approaches using diverse PLMs, optimized for scientific tasks, including SciBERT, SciNCL, and SPECTER2. We conduct hyperparameter tuning and investigate the impact of data augmentation from bibliographic databases such as OpenAlex, Semantic Scholar, and Crossref. Our results demonstrate that fine-tuning pre-trained models substantially enhances classification performance, with SPECTER2 emerging as the most accurate model. Moreover, enriching the dataset with additional metadata improves classification outcomes significantly, especially when integrating information from S2AG, OpenAlex and Crossref. Our best-performing approach achieves a weighted F1-score of 0.7415. Overall, our study contributes to the advancement of reliable automated systems for scholarly publication categorization, offering a potential solution to the laborious manual curation process, thereby facilitating researchers in efficiently locating relevant resources.