mlsanityData to 14 Oct 2026

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

arXiv 2211.13308 ↗First posted 23 Nov 2022cs.CL cs.AI cs.IRcited by 129

Not on the map. Why?

Abstract (arXiv)

Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Builds on 40 of its 81 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

40 of 81 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  12. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
28 more
  1. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  3. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  4. 2014Distributed Representations of Sentences and Documentscited by 678
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  7. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  8. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  9. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  10. 2022UL2: Unifying Language Learning Paradigmscited by 295
  11. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  12. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  13. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  14. 2020CORD-19: The COVID-19 Open Research Datasetcited by 180
  15. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  16. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  17. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  18. 2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
  19. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  20. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  21. 2018Word Mover's Embedding: From Word2Vec to Document Embeddingcited by 26
  22. 2021CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Examplecited by 20
  23. 2022Exploring the Role of Task Transferability in Large-Scale Multi-Task Learningcited by 17
  24. 2021S2AND: A Benchmark and Evaluation System for Author Name Disambiguationcited by 13
  25. 2022Specialized Document Embeddings for Aspect-based Similarity of Research Paperscited by 12
  26. 2021Contrastive Document Representation Learning with Graph Attention Networkscited by 6
  27. 2022Large-scale Evaluation of Transformer-based Article Encoders on the Task of Citation Recommendationcited by 3
  28. 2021TweetPap: A Dataset to Study the Social Media Discourse of Scientific Paperscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 29; 2025: 29; 2026: 55.

129 papers.

Sort:

By area:

  1. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  2. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  3. 2023The Semantic Scholar Open Data Platformcited by 118
  4. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  5. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
  6. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 41
  7. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  8. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  9. 2024CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Supportcited by 29
  10. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  11. 2024LLMs4OM: Matching Ontologies with Large Language Modelscited by 22
  12. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
38 more
  1. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 20
  2. 2024PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Paperscited by 18
  3. 2023Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understandingcited by 16
  4. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  5. 2026HindSight: Evaluating LLM-Generated Research Ideas via Future Impactcited by 16
  6. 2023ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Sciencecited by 15
  7. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  8. 2024HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Predictioncited by 15
  9. 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 14
  10. 2023SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Designcited by 13
  11. 2023Personalized Jargon Identification for Enhanced Interdisciplinary Communicationcited by 12
  12. 2024CancerLLM: A Large Language Model in Cancer Domaincited by 12
  13. 2024Taxonomy-guided Semantic Indexing for Academic Paper Searchcited by 12
  14. 2025Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedbackcited by 12
  15. 2023A Gold Standard Dataset for the Reviewer Assignment Problemcited by 11
  16. 2023Chain-of-Factors Paper-Reviewer Matchingcited by 11
  17. 2024PaECTER: Patent-level Representation Learning using Citation-informed Transformerscited by 11
  18. 2024Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale?cited by 11
  19. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 11
  20. 2025How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?cited by 10
  21. 2025Scientific Paper Retrieval with LLM-Guided Semantic-Based Rankingcited by 10
  22. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  23. 2024On-the-fly Definition Augmentation of LLMs for Biomedical NERcited by 9
  24. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  25. 2024PhysBERT: A Text Embedding Model for Physics Scientific Literaturecited by 8
  26. 2025Automatic Evaluation Metrics for Artificially Generated Scientific Researchcited by 8
  27. 2023Exploring the Landscape of Natural Language Processing Researchcited by 7
  28. 2023NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domaincited by 7
  29. 2024Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusionscited by 7
  30. 2025Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generationcited by 7
  31. 2023When Large Language Models Meet Citation: A Surveycited by 6
  32. 2023Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoningcited by 6
  33. 2024Acceleron: A Tool to Accelerate Research Ideationcited by 6
  34. 2024HDT: Hierarchical Document Transformercited by 6
  35. 2023CARE: Extracting Experimental Findings From Clinical Literaturecited by 5
  36. 2025CorpusStudio: Surfacing Emergent Patterns in a Corpus of Prior Work while Writingcited by 5
  37. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  38. 2025SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMscited by 5

50 most cited of 129 All 129 (JSON)