mlsanityData to 7 Oct 2026

Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

arXiv 2401.14019 ↗First posted 25 Jan 2024cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 7 of its 13 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

7 of 13 references.

References · 7

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  5. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  6. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  7. 2023Efficient Benchmarking of Language Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 7; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  2. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 25
  3. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  4. 2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 10
  5. 2024A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarioscited by 8
  6. 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 8
  7. 2024Ranking Large Language Models without Ground Truthcited by 7
  8. 2024Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applicationscited by 6
  9. 2025Are Bias Evaluation Methods Biased ?cited by 4
  10. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 3
  11. 2025Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judgescited by 2
  12. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 2
6 more
  1. 2025QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Modelscited by 1
  2. 2025Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentationcited by 1
  3. 2026EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluationcited by 1
  4. 2026Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleadingcited by 1
  5. 2025N2: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completioncited by 0
  6. 2026SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Modelscited by 0