# Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data

arXiv: https://arxiv.org/abs/2604.02031 · HTML: https://mlsanity.com/paper/2604.02031/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 30 of 32)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Online Batch Selection for Faster Training of Neural Networks](https://mlsanity.com/paper/1511.06343/) (2015)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [A Survey on Long-Tailed Visual Recognition](https://mlsanity.com/paper/2205.13775/) (2022)
- [Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentation](https://mlsanity.com/paper/2106.12387/) (2021)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [A Loss Function for Generative Neural Networks Based on Watson's Perceptual Model](https://mlsanity.com/paper/2006.15057/) (2020)
- [Improved anomaly detection by training an autoencoder with skip connections on images corrupted with Stain-shaped noise](https://mlsanity.com/paper/2008.12977/) (2020)
- [Exploring large scale public medical image datasets](https://mlsanity.com/paper/1907.12720/) (2019)
- [HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks](https://mlsanity.com/paper/2305.06382/) (2023)
- [Explicitly Minimizing the Blur Error of Variational Autoencoders](https://mlsanity.com/paper/2304.05939/) (2023)
- [Bridging Category-level and Instance-level Semantic Image Segmentation](https://mlsanity.com/paper/1605.06885/) (2016)
- [Training a Task-Specific Image Reconstruction Loss](https://mlsanity.com/paper/2103.14616/) (2021)
- [A step towards understanding why classification helps regression](https://mlsanity.com/paper/2308.10603/) (2023)
- [Hard Pixel Mining for Depth Privileged Semantic Segmentation](https://mlsanity.com/paper/1906.11437/) (2019)
- [Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems](https://mlsanity.com/paper/2410.01376/) (2024)
- [A Survey of Deep Long-Tail Classification Advancements](https://mlsanity.com/paper/2404.15593/) (2024)

## Abstract (arXiv)

Autoencoders can be challenged by spatially non-uniform sampling of image content. This is common in medical imaging, biology, and physics, where informative patterns occur rarely at specific image coordinates, as background dominates these locations in most samples, biasing reconstructions toward the majority appearance. In practice, autoencoders are biased toward dominant patterns resulting in the loss of fine-grained detail and causing blurred reconstructions for rare spatial inputs especially under spatial data imbalance. We address spatial imbalance by two complementary components: (i) self-entropy-based loss that upweights statistically uncommon spatial locations and (ii) Sample Propagation, a replay mechanism that selectively re-exposes the model to hard to reconstruct samples across batches during training. We benchmark existing data balancing strategies, originally developed for supervised classification, in the unsupervised reconstruction setting. Drawing on the limitations of these approaches, our method specifically targets spatial imbalance by encouraging models to focus on statistically rare locations, improving reconstruction consistency compared to existing baselines. We validate in a simulated dataset with controlled spatial imbalance conditions, and in three, uncontrolled, diverse real-world datasets spanning physical, biological, and astronomical domains. Our approach outperforms baselines on various reconstruction metrics, particularly under spatial imbalance distributions. These results highlight the importance of data representation in a batch and emphasize rare samples in unsupervised image reconstruction. We will make all code and related data available.