# Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

arXiv: https://arxiv.org/abs/2401.14019 · HTML: https://mlsanity.com/paper/2401.14019/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)

## Built on by (most cited first, 18 of 18)

- [OLMES: A Standard for Language Model Evaluations](https://mlsanity.com/paper/2406.08446/) (2024)
- [Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench](https://mlsanity.com/paper/2407.13696/) (2024)
- [DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation](https://mlsanity.com/paper/2503.01622/) (2025)
- [ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments](https://mlsanity.com/paper/2505.22169/) (2025)
- [BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks](https://mlsanity.com/paper/2410.12974/) (2024)
- [Ranking Large Language Models without Ground Truth](https://mlsanity.com/paper/2402.14860/) (2024)
- [A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios](https://mlsanity.com/paper/2408.01963/) (2024)
- [Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications](https://mlsanity.com/paper/2407.18990/) (2024)
- [Are Bias Evaluation Methods Biased ?](https://mlsanity.com/paper/2506.17111/) (2025)
- [HarnessEval-W: Agentifying the Evaluation of Visual Worlds](https://mlsanity.com/paper/2608.16859/) (2026)
- [Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results](https://mlsanity.com/paper/2606.14516/) (2026)
- [Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges](https://mlsanity.com/paper/2511.04478/) (2025)
- [QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models](https://mlsanity.com/paper/2512.08646/) (2025)
- [Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation](https://mlsanity.com/paper/2512.09577/) (2025)
- [EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation](https://mlsanity.com/paper/2602.18823/) (2026)
- [Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading](https://mlsanity.com/paper/2604.27637/) (2026)
- [N2: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completion](https://mlsanity.com/paper/2506.04166/) (2025)
- [SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models](https://mlsanity.com/paper/2605.19357/) (2026)

## Abstract (arXiv)

In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!