# Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face

arXiv: https://arxiv.org/abs/2302.14534 · HTML: https://mlsanity.com/paper/2302.14534/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://mlsanity.com/paper/2210.09984/) (2022)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [Shirtless and Dangerous: Quantifying Linguistic Signals of Gender Bias in an Online Fiction Writing Community](https://mlsanity.com/paper/1603.08832/) (2016)
- [Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset](https://mlsanity.com/paper/2007.07846/) (2020)
- [BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model](https://mlsanity.com/paper/2212.04960/) (2022)
- [Quote Erat Demonstrandum: A Web Interface for Exploring the Quotebank Corpus](https://mlsanity.com/paper/2207.03592/) (2022)

## Built on by (most cited first, 3 of 3)

- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Read, Look or Listen? What's Needed for Solving a Multimodal Dataset](https://mlsanity.com/paper/2307.04532/) (2023)
- [Indicative Summarization of Long Discussions](https://mlsanity.com/paper/2311.01882/) (2023)

## Abstract (arXiv)

We present Spacerini, a tool that integrates the Pyserini toolkit for reproducible information retrieval research with Hugging Face to enable the seamless construction and deployment of interactive search engines. Spacerini makes state-of-the-art sparse and dense retrieval models more accessible to non-IR practitioners while minimizing deployment effort. This is useful for NLP researchers who want to better understand and validate their research by performing qualitative analyses of training corpora, for IR researchers who want to demonstrate new retrieval models integrated into the growing Pyserini ecosystem, and for third parties reproducing the work of other researchers. Spacerini is open source and includes utilities for loading, preprocessing, indexing, and deploying search engines locally and remotely. We demonstrate a portfolio of 13 search engines created with Spacerini for different use cases.