# Boosting K-means for Big Data by Fusing Data Streaming with Global Optimization

arXiv: https://arxiv.org/abs/2410.14548 · HTML: https://mlsanity.com/paper/2410.14548/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [HG-means: A scalable hybrid genetic algorithm for minimum sum-of-squares clustering](https://mlsanity.com/paper/1804.09813/) (2018)
- [How to Use K-means for Big Data Clustering?](https://mlsanity.com/paper/2204.07485/) (2022)
- [Qualitative Properties of the Minimum Sum-of-Squares Clustering Problem](https://mlsanity.com/paper/1810.02057/) (2018)
- [High-Performance Hybrid Algorithm for Minimum Sum-of-Squares Clustering of Infinitely Tall Data](https://mlsanity.com/paper/2311.04517/) (2023)
- [Variable Landscape Search: A Novel Metaheuristic Paradigm for Unlocking Hidden Dimensions in Global Optimization](https://mlsanity.com/paper/2408.03895/) (2024)

## Abstract (arXiv)

K-means clustering is a cornerstone of data mining, but its efficiency deteriorates when confronted with massive datasets. To address this limitation, we propose a novel heuristic algorithm that leverages the Variable Neighborhood Search (VNS) metaheuristic to optimize K-means clustering for big data. Our approach is based on the sequential optimization of the partial objective function landscapes obtained by restricting the Minimum Sum-of-Squares Clustering (MSSC) formulation to random samples from the original big dataset. Within each landscape, systematically expanding neighborhoods of the currently best (incumbent) solution are explored by reinitializing all degenerate and a varying number of additional centroids. Extensive and rigorous experimentation on a large number of real-world datasets reveals that by transforming the traditional local search into a global one, our algorithm significantly enhances the accuracy and efficiency of K-means clustering in big data environments, becoming the new state of the art in the field.