# Pytrec_eval: An Extremely Fast Python Interface to trec_eval

arXiv: https://arxiv.org/abs/1805.01597 · HTML: https://mlsanity.com/paper/1805.01597/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2018.

## Built on by (most cited first, 30 of 46)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search](https://mlsanity.com/paper/2406.05013/) (2024)
- [Learning to Relate to Previous Turns in Conversational Search](https://mlsanity.com/paper/2306.02553/) (2023)
- [Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation](https://mlsanity.com/paper/2402.11827/) (2024)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers](https://mlsanity.com/paper/2406.10991/) (2024)
- [AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment](https://mlsanity.com/paper/2407.01965/) (2024)
- [How to Leverage Personal Textual Knowledge for Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2407.16192/) (2024)
- [Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search](https://mlsanity.com/paper/2407.20189/) (2024)
- [DAPR: A Benchmark on Document-Aware Passage Retrieval](https://mlsanity.com/paper/2305.13915/) (2023)
- [ConvSDG: Session Data Generation for Conversational Search](https://mlsanity.com/paper/2403.11335/) (2024)
- [A Reproducibility Study of PLAID](https://mlsanity.com/paper/2404.14989/) (2024)
- [mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval](https://mlsanity.com/paper/2501.19264/) (2025)
- [From Easy to Hard: A Dual Curriculum Learning Framework for Context-Aware Document Ranking](https://mlsanity.com/paper/2208.10226/) (2022)
- [Efficient and Asymptotically Unbiased Constrained Decoding for Large Language Models](https://mlsanity.com/paper/2504.09135/) (2025)
- [ConvMix: A Mixed-Criteria Data Augmentation Framework for Conversational Dense Retrieval](https://mlsanity.com/paper/2508.04001/) (2025)
- [Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking](https://mlsanity.com/paper/2210.10695/) (2022)
- [History-Aware Conversational Dense Retrieval](https://mlsanity.com/paper/2401.16659/) (2024)
- [Learning Job Title Representation from Job Description Aggregation Network](https://mlsanity.com/paper/2406.08055/) (2024)
- [DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs](https://mlsanity.com/paper/2406.17158/) (2024)
- [Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model](https://mlsanity.com/paper/2409.17745/) (2024)
- [LePaRD: A Large-Scale Dataset of Judges Citing Precedents](https://mlsanity.com/paper/2311.09356/) (2023)
- [Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search](https://mlsanity.com/paper/2505.14156/) (2025)
- [SPARQL-LLM: Real-Time SPARQL Query Generation from Natural Language Questions](https://mlsanity.com/paper/2512.14277/) (2025)
- [Conversational Query Reformulation with the Guidance of Retrieved Documents](https://mlsanity.com/paper/2407.12363/) (2024)
- [AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation](https://mlsanity.com/paper/2506.01381/) (2025)
- [URL: Universal Referential Knowledge Linking via Task-instructed Representation Compression](https://mlsanity.com/paper/2404.16248/) (2024)

## Abstract (arXiv)

We introduce pytrec_eval, a Python interface to the tree_eval information retrieval evaluation toolkit. pytrec_eval exposes the reference implementations of trec_eval within Python as a native extension. We show that pytrec_eval is around one order of magnitude faster than invoking trec_eval as a sub process from within Python. Compared to a native Python implementation of NDCG, pytrec_eval is twice as fast for practically-sized rankings. Finally, we demonstrate its effectiveness in an application where pytrec_eval is combined with Pyndri and the OpenAI Gym where query expansion is learned using Q-learning.