mlsanityData to 7 Oct 2026

SPECTER: Document-level Representation Learning using Citation-informed Transformers

arXiv 2004.07180 ↗First posted 15 Apr 2020cs.CLcited by 426

Not on the map. Why?

Cited by 42 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Representation learning is a critical ingredient for natural language processing systems. Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks. We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation. We show that SPECTER outperforms a variety of competitive baselines on the benchmark.

Key facts. Cited by 426 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 52; 2023: 70; 2024: 89; 2025: 104; 2026: 86.

426 papers.

Sort:

By area:

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  4. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  7. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  8. 2024Generative Representational Instruction Tuningcited by 227
  9. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  11. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  12. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
38 more
  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  2. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  3. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  4. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  5. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  6. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  7. 2023The Semantic Scholar Open Data Platformcited by 109
  8. 2022Task-aware Retrieval with Instructionscited by 107
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  11. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  12. 2024Making Text Embedders Few-Shot Learnerscited by 78
  13. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  14. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  15. 2024Autonomous LLM-driven research from data to human-verifiable research paperscited by 50
  16. 2023Patton: Language Model Pretraining on Text-Rich Networkscited by 49
  17. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  18. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  19. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  20. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  21. 2024Is Semantic Chunking Worth the Computational Cost?cited by 43
  22. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
  23. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  24. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  25. 2024IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedbackcited by 40
  26. 2024Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluationcited by 39
  27. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  28. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 38
  29. 2022Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Reviewcited by 37
  30. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  31. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  32. 2023Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Trainingcited by 35
  33. 2021Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distributioncited by 34
  34. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  35. 2023ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddingscited by 32
  36. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 29
  37. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  38. 2021Simplified Data Wrangling with ir_datasetscited by 27

50 most cited of 426 All 426 (JSON)