# ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers

arXiv: https://arxiv.org/abs/2505.08941 · HTML: https://mlsanity.com/paper/2505.08941/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [MultiSChuBERT: Effective Multimodal Fusion for Scholarly Document Quality Prediction](https://mlsanity.com/paper/2308.07971/) (2023)
- [CiMaTe: Citation Count Prediction Effectively Leveraging the Main Text](https://mlsanity.com/paper/2410.04404/) (2024)

## Built on by (most cited first, 1 of 1)

- [LLM-Metrics: Measuring Research Impact Through Large Language Model Memory](https://mlsanity.com/paper/2605.22176/) (2026)

## Abstract (arXiv)

Predicting the future citation rates of academic papers is an important step toward the automation of research evaluation and the acceleration of scientific progress. We present ForeCite, a simple but powerful framework to append pre-trained causal language models with a linear head for average monthly citation rate prediction. Adapting transformers for regression tasks, ForeCite achieves a test correlation of ρ = 0.826 on a curated dataset of 900K+ biomedical papers published between 2000 and 2024, a 27-point improvement over the previous state-of-the-art. Comprehensive scaling-law analysis reveals consistent gains across model sizes and data volumes, while temporal holdout experiments confirm practical robustness. Gradient-based saliency heatmaps suggest a potentially undue reliance on titles and abstract texts. These results establish a new state-of-the-art in forecasting the long-term influence of academic research and lay the groundwork for the automated, high-fidelity evaluation of scientific contributions.