mlsanityData to 7 Oct 2026

Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

arXiv 2403.19340 ↗First posted 28 Mar 2024cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to welcome community contribution. Additionally, we provide a concise, two-minute video demonstration of our system, illustrating its capabilities and implementation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

15 of 21 references.

References · 15

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  7. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  8. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  9. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  10. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  11. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 60
  12. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
3 more
  1. 2023Data Management For Training Large Language Models: A Surveycited by 17
  2. 2021Are Bias Mitigation Techniques for Deep Learning Effective?cited by 14
  3. 2023DMOps: Data Management Operation and Recipescited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of LLM × DATAcited by 4