# CiteBART: Learning to Generate Citations for Local Citation Recommendation

arXiv: https://arxiv.org/abs/2412.17534 · HTML: https://mlsanity.com/paper/2412.17534/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks](https://mlsanity.com/paper/1903.06464/) (2019)
- [Learning to Plan and Generate Text with Citations](https://mlsanity.com/paper/2404.03381/) (2024)
- [Predicting citation counts based on deep neural network learning techniques](https://mlsanity.com/paper/1809.04365/) (2018)
- [Local Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERT-based Reranking](https://mlsanity.com/paper/2112.01206/) (2021)
- [SChuBERT: Scholarly Document Chunks with BERT-encoding boost Citation Count Prediction](https://mlsanity.com/paper/2012.11740/) (2020)
- [Prototype-Based Interpretability for Legal Citation Prediction](https://mlsanity.com/paper/2305.16490/) (2023)
- [Predicting the Citations of Scholarly Paper](https://mlsanity.com/paper/2008.05013/) (2020)
- [SymTax: Symbiotic Relationship and Taxonomy Fusion for Effective Citation Recommendation](https://mlsanity.com/paper/2406.01606/) (2024)

## Built on by (most cited first, 6 of 6)

- [Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attribution](https://mlsanity.com/paper/2509.21557/) (2025)
- [Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild](https://mlsanity.com/paper/2603.17361/) (2026)
- [Talk2Ref: A Dataset for Reference Prediction from Scientific Talks](https://mlsanity.com/paper/2510.24478/) (2025)
- [RWGBench: Evaluating Scholarly Positioning in Related Work Generation](https://mlsanity.com/paper/2606.24894/) (2026)
- [ReCite: Agentic Reasoning for Faithful Citation](https://mlsanity.com/paper/2609.09156/) (2026)
- [ATTRICITE: Training an Open 4B Model for Citation Recovery toward Faithful Attribution](https://mlsanity.com/paper/2609.14248/) (2026)

## Abstract (arXiv)

Local citation recommendation (LCR) suggests a set of papers for a citation placeholder within a given context. The task has evolved as generative approaches have become more promising than the traditional pre-fetch and re-rank-based state-of-the-art approaches. This paper introduces citation-specific pre-training within an encoder-decoder architecture, where author-date citation tokens are masked to learn to reconstruct them to fulfill LCR. There are two variants for this pre-training. In the local context-only base scheme (CiteBART-Base), the citation token in a local context is masked to learn to predict the citation. The global version (CiteBART-Global) extends the local context with the citing paper's title and abstract to enrich the learning signal. CiteBART-Global achieves state-of-the-art performance on LCR benchmarks except for the FullTextPeerRead dataset, which is quite small to see the advantage of generative pre-training. The effect is significant in the larger benchmarks, e.g., Refseer and ArXiv., with the Refseer benchmark-trained model emerging as the best-performing model. We perform comprehensive experiments, including an ablation study, a qualitative analysis, and a taxonomy of hallucinations with detailed statistics. Our analyses confirm that CiteBART-Global has a cross-dataset generalization capability; the macro hallucination rate (MaHR) at the top-3 predictions is 4%, and when the ground-truth is in the top-k prediction list, the hallucination tendency in the other predictions drops significantly.