# On Big Data Benchmarking

arXiv: https://arxiv.org/abs/1402.5194 · HTML: https://mlsanity.com/paper/1402.5194/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2014.

## Built on by (most cited first, 3 of 3)

- [LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation](https://mlsanity.com/paper/2511.14531/) (2025)
- [ASTRA-QA: A Benchmark for Abstract Question Answering over Documents](https://mlsanity.com/paper/2605.10168/) (2026)
- [MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification](https://mlsanity.com/paper/2608.22460/) (2026)

## Abstract (arXiv)

Big data systems address the challenges of capturing, storing, managing, analyzing, and visualizing big data. Within this context, developing benchmarks to evaluate and compare big data systems has become an active topic for both research and industry communities. To date, most of the state-of-the-art big data benchmarks are designed for specific types of systems. Based on our experience, however, we argue that considering the complexity, diversity, and rapid evolution of big data systems, for the sake of fairness, big data benchmarks must include diversity of data and workloads. Given this motivation, in this paper, we first propose the key requirements and challenges in developing big data benchmarks from the perspectives of generating data with 4V properties (i.e. volume, velocity, variety and veracity) of big data, as well as generating tests with comprehensive workloads for big data systems. We then present the methodology on big data benchmarking designed to address these challenges. Next, the state-of-the-art are summarized and compared, following by our vision for future research directions.