# Challenges of Big Data Analysis

arXiv: https://arxiv.org/abs/1308.1479 · HTML: https://mlsanity.com/paper/1308.1479/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2013.

## Built on by (most cited first, 30 of 47)

- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset](https://mlsanity.com/paper/2110.00715/) (2021)
- [Environment Invariant Linear Least Squares](https://mlsanity.com/paper/2303.03092/) (2023)
- [Deep Learning for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2503.10198/) (2025)
- [Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneity](https://mlsanity.com/paper/2306.06291/) (2023)
- [Spurious Correlations and Where to Find Them](https://mlsanity.com/paper/2308.11043/) (2023)
- [Mitigating the Effect of Incidental Correlations on Part-based Learning](https://mlsanity.com/paper/2310.00377/) (2023)
- [Data-Driven Minimax Optimization with Expectation Constraints](https://mlsanity.com/paper/2202.07868/) (2022)
- [Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation](https://mlsanity.com/paper/2410.05401/) (2024)
- [Entangled Mean Estimation in High-Dimensions](https://mlsanity.com/paper/2501.05425/) (2025)
- [Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos](https://mlsanity.com/paper/2309.04236/) (2023)
- [Semantic-Preserving Feature Partitioning for Multi-View Ensemble Learning](https://mlsanity.com/paper/2401.06251/) (2024)
- [Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation](https://mlsanity.com/paper/2406.18562/) (2024)
- [Enhancing Machine Learning Performance through Intelligent Data Quality Assessment: An Unsupervised Data-centric Framework](https://mlsanity.com/paper/2502.13198/) (2025)
- [Sequential Estimation under Multiple Resources: a Bandit Point of View](https://mlsanity.com/paper/2109.14703/) (2021)
- [On How AI Needs to Change to Advance the Science of Drug Discovery](https://mlsanity.com/paper/2212.12560/) (2022)
- [Predictive Heterogeneity: Measures and Applications](https://mlsanity.com/paper/2304.00305/) (2023)
- [Exploring and Exploiting Data Heterogeneity in Recommendation](https://mlsanity.com/paper/2305.15431/) (2023)
- [A Compact Implicit Neural Representation for Efficient Storage of Massive 4D Functional Magnetic Resonance Imaging](https://mlsanity.com/paper/2312.00082/) (2023)
- [A Kernel-Based Neural Network Test for High-dimensional Sequencing Data Analysis](https://mlsanity.com/paper/2312.02850/) (2023)
- [A Combination Model Based on Sequential General Variational Mode Decomposition Method for Time Series Prediction](https://mlsanity.com/paper/2406.03157/) (2024)
- [Factor Adjusted Spectral Clustering for Mixture Models](https://mlsanity.com/paper/2408.12564/) (2024)
- [A Bilevel Optimization Framework for Imbalanced Data Classification](https://mlsanity.com/paper/2410.11171/) (2024)
- [Feature Selection Based on Wasserstein Distance](https://mlsanity.com/paper/2411.07217/) (2024)
- [An AI-Based Public Health Data Monitoring System](https://mlsanity.com/paper/2506.04429/) (2025)
- [Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models](https://mlsanity.com/paper/2508.11460/) (2025)
- [Contrastive Dimension Reduction: A Systematic Review](https://mlsanity.com/paper/2510.11847/) (2025)
- [To do or not to do: finding causal relations in smart homes](https://mlsanity.com/paper/2105.10058/) (2021)
- [Pre-processing with Orthogonal Decompositions for High-dimensional Explanatory Variables](https://mlsanity.com/paper/2106.09071/) (2021)
- [Prediction of terephthalic acid (TPA) yield in aqueous hydrolysis of polyethylene terephthalate (PET)](https://mlsanity.com/paper/2201.12657/) (2022)

## Abstract (arXiv)

Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity, and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This article give overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasis on the viability of the sparsest solution in high-confidence set and point out that exogeneous assumptions in most statistical methods for Big Data can not be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.