# Blocks Architecture (BloArk): Efficient, Cost-Effective, and Incremental Dataset Architecture for Wikipedia Revision History

arXiv: https://arxiv.org/abs/2410.04410 · HTML: https://mlsanity.com/paper/2410.04410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 6 of 6)

- [Learning To Split and Rephrase From Wikipedia Edit History](https://mlsanity.com/paper/1808.09468/) (2018)
- [MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score](https://mlsanity.com/paper/2210.05875/) (2022)
- [Split and Rephrase with Large Language Models](https://mlsanity.com/paper/2312.11075/) (2023)
- [TimeMachine: Timeline Generation for Knowledge-Base Entities](https://mlsanity.com/paper/1502.04662/) (2015)
- [Wikidated 1.0: An Evolving Knowledge Graph Dataset of Wikidata's Revision History](https://mlsanity.com/paper/2112.05003/) (2021)
- [Leveraging Wikidata's edit history in knowledge graph refinement tasks](https://mlsanity.com/paper/2210.15495/) (2022)

## Abstract (arXiv)

Wikipedia (Wiki) is one of the most widely used and publicly available resources for natural language processing (NLP) applications. Wikipedia Revision History (WikiRevHist) shows the order in which edits were made to any Wiki page since its first modification. While the most up-to-date Wiki has been widely used as a training source, WikiRevHist can also be valuable resources for NLP applications. However, there are insufficient tools available to process WikiRevHist without having substantial computing resources, making additional customization, and spending extra time adapting others' works. Therefore, we report Blocks Architecture (BloArk), an efficiency-focused data processing architecture that reduces running time, computing resource requirements, and repeated works in processing WikiRevHist dataset. BloArk consists of three parts in its infrastructure: blocks, segments, and warehouses. On top of that, we build the core data processing pipeline: builder and modifier. The BloArk builder transforms the original WikiRevHist dataset from XML syntax into JSON Lines (JSONL) format for improving the concurrent and storage efficiency. The BloArk modifier takes previously-built warehouses to operate incremental modifications for improving the utilization of existing databases and reducing the cost of reusing others' works. In the end, BloArk can scale up easily in both processing Wikipedia Revision History and incrementally modifying existing dataset for downstream NLP use cases. The source code, documentations, and example usages are publicly available online and open-sourced under GPL-2.0 license.