# When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer

arXiv: https://arxiv.org/abs/2412.20323 · HTML: https://mlsanity.com/paper/2412.20323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 29 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014)
- [Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets](https://mlsanity.com/paper/1406.7343/) (2014)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [A Case Study Competition Among Methods for Analyzing Large Spatial Data](https://mlsanity.com/paper/1710.05013/) (2017)
- [Neural Methods for Amortized Inference](https://mlsanity.com/paper/2404.12484/) (2024)
- [Likelihood-Free Parameter Estimation with Neural Bayes Estimators](https://mlsanity.com/paper/2208.12942/) (2022)
- [Neural Networks for Parameter Estimation in Intractable Models](https://mlsanity.com/paper/2107.14346/) (2021)
- [Permutation and Grouping Methods for Sharpening Gaussian Process Approximations](https://mlsanity.com/paper/1609.05372/) (2016)
- [Fast covariance parameter estimation of spatial Gaussian process models using neural networks](https://mlsanity.com/paper/2012.15339/) (2020)
- [A multi-resolution approximation for massive spatial datasets](https://mlsanity.com/paper/1507.04789/) (2015)
- [Neural Bayes Estimators for Irregular Spatial Data using Graph Neural Networks](https://mlsanity.com/paper/2310.02600/) (2023)
- [Gaussian Process Learning via Fisher Scoring of Vecchia's Approximation](https://mlsanity.com/paper/1905.08374/) (2019)
- [A Distributed and Integrated Method of Moments for High-Dimensional Correlated Data Analysis](https://mlsanity.com/paper/1910.02986/) (2019)
- [Towards black-box parameter estimation](https://mlsanity.com/paper/2303.15041/) (2023)
- [Demonstrating the power and flexibility of variational assumptions for amortized neural posterior estimation in environmental applications](https://mlsanity.com/paper/2404.10899/) (2024)
- [A class of multi-resolution approximations for large spatial datasets](https://mlsanity.com/paper/1710.08976/) (2017)
- [Distributed Inference for Spatial Extremes Modeling in High Dimensions](https://mlsanity.com/paper/2204.14165/) (2022)
- [Joint integrative analysis of multiple data sources with correlated vector outcomes](https://mlsanity.com/paper/2011.14996/) (2020)
- [Functional Regression with Intensively Measured Longitudinal Outcomes: A New Lens through Data Partitioning](https://mlsanity.com/paper/2207.13014/) (2022)
- [Distributed model building and recursive integration for big spatial data modeling](https://mlsanity.com/paper/2305.15951/) (2023)
- [Amortized Bayesian Local Interpolation NetworK: Fast covariance parameter estimation for Gaussian Processes](https://mlsanity.com/paper/2411.06324/) (2024)

## Abstract (arXiv)

Black-box methods such as deep neural networks are exceptionally fast at obtaining point estimates of model parameters due to their amortisation of the loss function computation, but are currently restricted to settings for which simulating training data is inexpensive. When simulating data is computationally expensive, both the training and uncertainty quantification, which typically relies on a parametric bootstrap, become intractable. We propose a black-box divide-and-conquer estimation and inference framework when data simulation is computationally expensive that trains a black-box estimation method on a partition of the multivariate data domain, estimates and bootstraps on the partitioned data, and combines estimates and inferences across data partitions. Through the divide step, only small training data need be simulated, substantially accelerating the training. Further, the estimation and bootstrapping can be conducted in parallel across multiple computing nodes to further speed up the procedure. Finally, the conquer step accounts for any dependence between data partitions through a statistically and computationally efficient weighted average. We illustrate the implementation of our framework in high-dimensional spatial settings with Gaussian and max-stable processes. Applications to modeling extremal temperature data from both a climate model and observations from the National Oceanic and Atmospheric Administration highlight the feasibility of estimation and inference of max-stable process parameters with tens of thousands of locations.