# Using a Power Law Distribution to describe Big Data

arXiv: https://arxiv.org/abs/1509.00504 · HTML: https://mlsanity.com/paper/1509.00504/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2015.

## Built on by (most cited first, 2 of 2)

- [Developing a Series of AI Challenges for the United States Department of the Air Force](https://mlsanity.com/paper/2207.07033/) (2022)
- [Interpretable Transformation and Analysis of Timelines through Learning via Surprisability](https://mlsanity.com/paper/2503.04502/) (2025)

## Abstract (arXiv)

The gap between data production and user ability to access, compute and produce meaningful results calls for tools that address the challenges associated with big data volume, velocity and variety. One of the key hurdles is the inability to methodically remove expected or uninteresting elements from large data sets. This difficulty often wastes valuable researcher and computational time by expending resources on uninteresting parts of data. Social sensors, or sensors which produce data based on human activity, such as Wikipedia, Twitter, and Facebook have an underlying structure which can be thought of as having a Power Law distribution. Such a distribution implies that few nodes generate large amounts of data. In this article, we propose a technique to take an arbitrary dataset and compute a power law distributed background model that bases its parameters on observed statistics. This model can be used to determine the suitability of using a power law or automatically identify high degree nodes for filtering and can be scaled to work with big data.