Uncertainty Quantification using Simulation Output: Batching as an Inferential Device

We present batching as an omnibus device for uncertainty quantification using simulation output. We consider the classical context of a simulationist performing uncertainty quantification on an estimator θn (of an unknown fixed quantity θ) using only the output data (Y1,Y2,ldots,Yn) gathered from a simulation. By uncertainty quantification, we mean approximating the sampling distribution of the error θn-θ toward: (A) estimating an assessment functional psi, e.g., bias, variance, or quantile; or (B) constructing a (1-α)-confidence region on θ. We argue that batching is a remarkably simple and effective device for this purpose, and is especially suited for handling dependent output data such as what one frequently encounters in simulation contexts. We demonstrate that if the number of batches and the extent of their overlap are chosen appropriately, batching retains bootstrap's attractive theoretical properties of strong consistency and higher-order accuracy. For constructing confidence regions, we characterize two limiting distributions associated with a Studentized statistic. Our extensive numerical experience confirms theoretical insight, especially about the effects of batch size and batch overlap.