# Adapting to Skew: Imputing Spatiotemporal Urban Data with 3D Partial Convolutions and Biased Masking

arXiv: https://arxiv.org/abs/2301.04233 · HTML: https://mlsanity.com/paper/2301.04233/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 67 references that are in the map.
- First posted on arXiv 10 Jan 2023.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [Learning Traffic as Images: A Deep Convolutional Neural Network for Large-Scale Transportation Network Speed Prediction](https://mlsanity.com/paper/1701.04245/) (2017)
- [Region Normalization for Image Inpainting](https://mlsanity.com/paper/1911.10375/) (2019)
- [Spatiotemporal Recurrent Convolutional Networks for Traffic Prediction in Transportation Networks](https://mlsanity.com/paper/1705.02699/) (2017)
- [Predicting Citywide Crowd Flows Using Deep Spatio-Temporal Residual Networks](https://mlsanity.com/paper/1701.02543/) (2017)
- [Deep Sequence Learning with Auxiliary Information for Traffic Prediction](https://mlsanity.com/paper/1806.07380/) (2018)
- [Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network](https://mlsanity.com/paper/1802.08369/) (2018)
- [Texture Memory-Augmented Deep Patch-Based Image Inpainting](https://mlsanity.com/paper/2009.13240/) (2020)
- [Learning Oracle Attention for High-fidelity Face Completion](https://mlsanity.com/paper/2003.13903/) (2020)
- [FairST: Equitable Spatial and Temporal Demand Prediction for New Mobility Systems](https://mlsanity.com/paper/1907.03827/) (2019)

## Built on by (most cited first, 1 of 1)

- [Fragments to Facts: Partial-Information Fragment Inference from LLMs](https://mlsanity.com/paper/2505.13819/) (2025)

## Abstract (arXiv)

We adapt image inpainting techniques to impute large, irregular missing regions in urban settings characterized by sparsity, variance in both space and time, and anomalous events. Missing regions in urban data can be caused by sensor or software failures, data quality issues, interference from weather events, incomplete data collection, or varying data use regulations; any missing data can render the entire dataset unusable for downstream applications. To ensure coverage and utility, we adapt computer vision techniques for image inpainting to operate on 3D histograms (2D space + 1D time) commonly used for data exchange in urban settings. Adapting these techniques to the spatiotemporal setting requires handling skew: urban data tend to follow population density patterns (small dense regions surrounded by large sparse areas); these patterns can dominate the learning process and fool the model into ignoring local or transient effects. To combat skew, we 1) train simultaneously in space and time, and 2) focus attention on dense regions by biasing the masks used for training to the skew in the data. We evaluate the core model and these two extensions using the NYC taxi data and the NYC bikeshare data, simulating different conditions for missing data. We show that the core model is effective qualitatively and quantitatively, and that biased masking during training reduces error in a variety of scenarios. We also articulate a tradeoff in varying the number of timesteps per training sample: too few timesteps and the model ignores transient events; too many timesteps and the model is slow to train with limited performance gain.