# OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining

arXiv: https://arxiv.org/abs/2402.15810 · HTML: https://mlsanity.com/paper/2402.15810/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 78 references that are in the map.
- First posted on arXiv 24 Feb 2024.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)

## Built on by (most cited first, 14 of 14)

- [AutoMind: Adaptive Knowledgeable Agent for Automated Data Science](https://mlsanity.com/paper/2506.10974/) (2025)
- [LLMs can Realize Combinatorial Creativity: Generating Creative Ideas via LLMs for Scientific Research](https://mlsanity.com/paper/2412.14141/) (2024)
- [RevGNN: Negative Sampling Enhanced Contrastive Graph Learning for Academic Reviewer Recommendation](https://mlsanity.com/paper/2407.20684/) (2024)
- [GuARD: Effective Anomaly Detection through a Text-Rich and Graph-Informed Language Model](https://mlsanity.com/paper/2412.03930/) (2024)
- [Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse Domains](https://mlsanity.com/paper/2412.08937/) (2024)
- [OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation](https://mlsanity.com/paper/2602.08896/) (2026)
- [Text-Driven Neural Collaborative Filtering Model for Paper Source Tracing](https://mlsanity.com/paper/2407.17722/) (2024)
- [LLM-Powered Ensemble Learning for Paper Source Tracing: A GPU-Free Approach](https://mlsanity.com/paper/2409.09383/) (2024)
- [ResearchArcade: Graph Interface for Academic Tasks](https://mlsanity.com/paper/2511.22036/) (2025)
- [Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression](https://mlsanity.com/paper/2601.08187/) (2026)
- [Explaining Synergistic Effects in Social Recommendations](https://mlsanity.com/paper/2601.18151/) (2026)
- [Cross-Source Reasoning-based Correction for Author Name Disambiguation](https://mlsanity.com/paper/2606.08617/) (2026)
- [AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs](https://mlsanity.com/paper/2607.20498/) (2026)
- [SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators](https://mlsanity.com/paper/2608.07641/) (2026)

## Abstract (arXiv)

With the rapid proliferation of scientific literature, versatile academic knowledge services increasingly rely on comprehensive academic graph mining. Despite the availability of public academic graphs, benchmarks, and datasets, these resources often fall short in multi-aspect and fine-grained annotations, are constrained to specific task types and domains, or lack underlying real academic graphs. In this paper, we present OAG-Bench, a comprehensive, multi-aspect, and fine-grained human-curated benchmark based on the Open Academic Graph (OAG). OAG-Bench covers 10 tasks, 20 datasets, 70+ baselines, and 120+ experimental results to date. We propose new data annotation strategies for certain tasks and offer a suite of data pre-processing codes, algorithm implementations, and standardized evaluation protocols to facilitate academic graph mining. Extensive experiments reveal that even advanced algorithms like large language models (LLMs) encounter difficulties in addressing key challenges in certain tasks, such as paper source tracing and scholar profiling. We also introduce the Open Academic Graph Challenge (OAG-Challenge) to encourage community input and sharing. We envisage that OAG-Bench can serve as a common ground for the community to evaluate and compare algorithms in academic graph mining, thereby accelerating algorithm development and advancement in this field. OAG-Bench is accessible at https://www.aminer.cn/data/.