mlsanityData to 7 Oct 2026

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

arXiv 2102.10073 ↗First posted 19 Feb 2021cs.IRcited by 60

Not on the map. Why?

Abstract (arXiv)

Pyserini is an easy-to-use Python toolkit that supports replicable IR research by providing effective first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. We also describe how our group has built a culture of replicability through shared norms and tools that enable rigorous automated testing.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 6; 2024: 8; 2025: 13; 2026: 10.

60 papers.

Sort:

By area:

  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  3. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  4. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  5. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  6. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  7. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  8. 2021Learning to Retrieve Passages without Supervisioncited by 46
  9. 2021A Replication Study of Dense Passage Retrievercited by 37
  10. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generatorcited by 34
  11. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  12. 2023WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpuscited by 29
38 more
  1. 2021Simplified Data Wrangling with ir_datasetscited by 27
  2. 2022Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrievalcited by 26
  3. 2021Yes, BM25 is a Strong Baseline for Legal Case Retrievalcited by 24
  4. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
  5. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  6. 2022Improving Biomedical Information Retrieval with Neural Retrieverscited by 20
  7. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  8. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversationcited by 19
  9. 2022Saving Dense Retriever from Shortcut Dependency in Conversational Searchcited by 18
  10. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17
  11. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  12. 2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
  13. 2022Improving Wikipedia Verifiability with AIcited by 11
  14. 2022Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verificationcited by 11
  15. 2025DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillationcited by 11
  16. 2025CSPLADE: Learned Sparse Retrieval with Causal Language Modelscited by 10
  17. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Levelcited by 10
  18. 2021Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answeringcited by 8
  19. 20231-PAGER: One Pass Answer Generation and Evidence Retrievalcited by 7
  20. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inferencecited by 5
  21. 2026Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Taskscited by 5
  22. 2025An Efficient Rubric-based Generative Verifier for Search-Augmented LLMscited by 4
  23. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasetscited by 4
  24. 2023Divide & Conquer for Entailment-aware Multi-hop Evidence Retrievalcited by 3
  25. 2025Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimatecited by 3
  26. 2025Rethinking On-policy Optimization for Query Augmentationcited by 3
  27. 2022XLMRQA: Open-Domain Question Answering on Vietnamese Wikipedia-based Textual Knowledge Sourcecited by 2
  28. 2022Diverse Multi-Answer Retrieval with Determinantal Point Processescited by 2
  29. 2024An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenderscited by 2
  30. 2026Evaluating LLMs on Real-World Software Performance Optimizationcited by 2
  31. 2021Towards Universal Dense Retrieval for Open-domain Question Answeringcited by 1
  32. 2022Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systemscited by 1
  33. 2022MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answeringcited by 1
  34. 2022NeuralSearchX: Serving a Multi-billion-parameter Reranker for Multilingual Metasearch at a Low Costcited by 1
  35. 2026BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigmscited by 1
  36. 2022To Tune or Not To Tune? Zero-shot Models for Legal Case Entailmentcited by 0
  37. 2023Cross Encoding as Augmentation: Towards Effective Educational Text Classificationcited by 0
  38. 2025MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queriescited by 0

50 most cited of 60 All 60 (JSON)