Mind the Residual Gap: Probabilistic Downscaling under Real-World Bias

Probabilistic downscaling models the conditional distribution of high-resolution fields given coarse inputs and is a core challenge in atmospheric science, climate modeling, and other multiscale physical systems. A widely used paradigm decomposes the problem into a deterministic mean predictor followed by a stochastic residual generator. While effective in idealized settings, this mean-residual approach frequently produces biased and underdispersive ensembles in real-world applications. We identify a fundamental source of this failure: residual target misspecification, where the residual distribution induced during training systematically differs from the correction distribution required at test time, a mismatch amplified by downscaling bias. We theoretically connect this misspecification to underdispersion through the spread-skill ratio and show that conditional residual distribution matching controls both residual-energy and conditional-mean mismatch. Motivated by this analysis, we introduce ReMatch (Residual Distribution Matching), which aligns training residual targets toward a held-out calibration regime via optimal transport. On a controlled synthetic benchmark with varying bias levels and a real-world HRRR-ERA5 wind-field downscaling task, ReMatch substantially reduces underdispersion, improves calibration, and outperforms strong mean-residual and super-resolution baselines. Our code is available at https://github.com/sdean-group/ReMatch.git.