# RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

arXiv: https://arxiv.org/abs/2608.27394 · HTML: https://mlsanity.com/paper/2608.27394/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 107 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [Jasper and Stella: distillation of SOTA embedding models](https://mlsanity.com/paper/2412.19048/) (2024)
- [BM25S: Orders of magnitude faster lexical search via eager sparse scoring](https://mlsanity.com/paper/2407.03618/) (2024)
- [ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition](https://mlsanity.com/paper/2503.21248/) (2025)
- [Mining Discourse Markers for Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/1903.11850/) (2019)
- [SciNLI: A Corpus for Natural Language Inference on Scientific Text](https://mlsanity.com/paper/2203.06728/) (2022)
- [Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora](https://mlsanity.com/paper/1806.03191/) (2018)
- [AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies](https://mlsanity.com/paper/2402.12370/) (2024)
- [MIR: Methodology Inspiration Retrieval for Scientific Research Problems](https://mlsanity.com/paper/2506.00249/) (2025)
- [HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance](https://mlsanity.com/paper/2506.12937/) (2025)
- [MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference](https://mlsanity.com/paper/2404.08066/) (2024)
- [OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries](https://mlsanity.com/paper/2605.06235/) (2026)
- [Scaling Creative Inspiration with Fine-Grained Functional Aspects of Ideas](https://mlsanity.com/paper/2102.09761/) (2021)
- [CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation](https://mlsanity.com/paper/2505.20779/) (2025)
- [Finding your MUSE: Mining Unexpected Solutions Engine](https://mlsanity.com/paper/2509.05072/) (2025)

## Abstract (arXiv)

Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction---zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision---previously used only for classification---to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers and LLM reranking helps further, but much room for improvement remains. Researchers blind to condition found the fine-tuned retriever's candidates more useful as inspirations. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues in scientific inspiration retrieval. We release full data, code, models, and annotations.