# CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example

arXiv: https://arxiv.org/abs/2103.12906 · HTML: https://mlsanity.com/paper/2103.12906/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 81 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 27 of 27)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [Pretrained Language Models for Sequential Sentence Classification](https://mlsanity.com/paper/1909.04054/) (2019)
- [Content-Based Citation Recommendation](https://mlsanity.com/paper/1802.08301/) (2018)
- [Citation Recommendation: Approaches and Datasets](https://mlsanity.com/paper/2002.06961/) (2020)
- [TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection](https://mlsanity.com/paper/2005.04474/) (2020)
- [Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality](https://mlsanity.com/paper/1801.00644/) (2018)
- [Aspect-based Document Similarity for Research Papers](https://mlsanity.com/paper/2010.06395/) (2020)
- [Syntactic Search by Example](https://mlsanity.com/paper/2006.03010/) (2020)
- [SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific search](https://mlsanity.com/paper/2005.12668/) (2020)
- [Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification](https://mlsanity.com/paper/2101.07124/) (2021)
- [Semantic query-by-example speech search using visual grounding](https://mlsanity.com/paper/1904.07078/) (2019)
- [Selecting Informative Contexts Improves Language Model Finetuning](https://mlsanity.com/paper/2005.00175/) (2020)
- [Interactive Extractive Search over Biomedical Corpora](https://mlsanity.com/paper/2006.04148/) (2020)
- [Learning Disentangled Representations of Texts with Application to Biomedical Abstracts](https://mlsanity.com/paper/1804.07212/) (2018)
- [Overview and Results: CL-SciSumm Shared Task 2019](https://mlsanity.com/paper/1907.09854/) (2019)
- [PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge](https://mlsanity.com/paper/2010.03725/) (2020)
- [Contextual Document Similarity for Content-based Literature Recommender Systems](https://mlsanity.com/paper/2008.00202/) (2020)
- [A New Neural Search and Insights Platform for Navigating and Organizing AI Research](https://mlsanity.com/paper/2011.00061/) (2020)
- [Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning](https://mlsanity.com/paper/1910.13339/) (2019)
- [Effective Distributed Representations for Academic Expert Search](https://mlsanity.com/paper/2010.08269/) (2020)
- [Will This Idea Spread Beyond Academia? Understanding Knowledge Transfer of Scientific Concepts across Text Corpora](https://mlsanity.com/paper/2010.06657/) (2020)
- [Aspect-based Academic Search using Domain-specific KB](https://mlsanity.com/paper/2001.10781/) (2020)

## Built on by (most cited first, 20 of 20)

- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity](https://mlsanity.com/paper/2111.08366/) (2021)
- [Large Language Model Augmented Narrative Driven Recommendations](https://mlsanity.com/paper/2306.02250/) (2023)
- [Literature-Grounded Novelty Assessment of Scientific Ideas](https://mlsanity.com/paper/2506.22026/) (2025)
- [PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Papers](https://mlsanity.com/paper/2403.02939/) (2024)
- [SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts](https://mlsanity.com/paper/2104.08809/) (2021)
- [Taxonomy-guided Semantic Indexing for Academic Paper Search](https://mlsanity.com/paper/2410.19218/) (2024)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)
- [Scientific Paper Retrieval with LLM-Guided Semantic-Based Ranking](https://mlsanity.com/paper/2505.21815/) (2025)
- [Augmenting Scientific Creativity with Retrieval across Knowledge Domains](https://mlsanity.com/paper/2206.01328/) (2022)
- [Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation](https://mlsanity.com/paper/2502.11181/) (2025)
- [CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrieval](https://mlsanity.com/paper/2505.13757/) (2025)
- [Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark](https://mlsanity.com/paper/2411.18831/) (2024)
- [Large-scale Evaluation of Transformer-based Article Encoders on the Task of Citation Recommendation](https://mlsanity.com/paper/2209.05452/) (2022)
- [Multi-Facet Blending for Faceted Query-by-Example Retrieval](https://mlsanity.com/paper/2412.01443/) (2024)
- [How Data Scientists Review the Scholarly Literature](https://mlsanity.com/paper/2301.03774/) (2023)
- [PaperRegister: Boosting Flexible-grained Paper Search via Hierarchical Register Indexing](https://mlsanity.com/paper/2508.11116/) (2025)
- [Improving Scientific Document Retrieval with Academic Concept Index](https://mlsanity.com/paper/2601.00567/) (2026)
- [Beyond Single-Score Ranking: Facet-Aware Reranking for Controllable Diversity in Paper Recommendation](https://mlsanity.com/paper/2604.16329/) (2026)
- [ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research](https://mlsanity.com/paper/2610.02202/) (2026)

## Abstract (arXiv)

Query by Example is a well-known information retrieval task in which a document is chosen by the user as the search query and the goal is to retrieve relevant documents from a large collection. However, a document often covers multiple aspects of a topic. To address this scenario we introduce the task of faceted Query by Example in which users can also specify a finer grained aspect in addition to the input query document. We focus on the application of this task in scientific literature search. We envision models which are able to retrieve scientific papers analogous to a query scientific paper along specifically chosen rhetorical structure elements as one solution to this problem. In this work, the rhetorical structure elements, which we refer to as facets, indicate objectives, methods, or results of a scientific paper. We introduce and describe an expert annotated test collection to evaluate models trained to perform this task. Our test collection consists of a diverse set of 50 query documents in English, drawn from computational linguistics and machine learning venues. We carefully follow the annotation guideline used by TREC for depth-k pooling (k = 100 or 250) and the resulting data collection consists of graded relevance scores with high annotation agreement. State of the art models evaluated on our dataset show a significant gap to be closed in further work. Our dataset may be accessed here: https://github.com/iesl/CSFCube