# MOSAIC: Masked Optimisation with Selective Attention for Image Reconstruction

arXiv: https://arxiv.org/abs/2306.00906 · HTML: https://mlsanity.com/paper/2306.00906/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [AMP-Net: Denoising based Deep Unfolding for Compressive Image Sensing](https://mlsanity.com/paper/2004.10078/) (2020)
- [Learned D-AMP: Principled Neural Network based Compressive Image Recovery](https://mlsanity.com/paper/1704.06625/) (2017)
- [ISTA-Net++: Flexible Deep Unfolding Network for Compressive Sensing](https://mlsanity.com/paper/2103.11554/) (2021)
- [Content-aware Scalable Deep Compressed Sensing](https://mlsanity.com/paper/2207.09313/) (2022)
- [Band-limited Training and Inference for Convolutional Neural Networks](https://mlsanity.com/paper/1911.09287/) (2019)
- [FSOINet: Feature-Space Optimization-Inspired Network for Image Compressive Sensing](https://mlsanity.com/paper/2204.05503/) (2022)

## Abstract (arXiv)

Compressive sensing (CS) reconstructs images from sub-Nyquist measurements by solving a sparsity-regularized inverse problem. Traditional CS solvers use iterative optimizers with hand crafted sparsifiers, while early data-driven methods directly learn an inverse mapping from the low-dimensional measurement space to the original image space. The latter outperforms the former, but is restrictive to a pre-defined measurement domain. More recent, deep unrolling methods combine traditional proximal gradient methods and data-driven approaches to iteratively refine an image approximation. To achieve higher accuracy, it has also been suggested to learn both the sampling matrix, and the choice of measurement vectors adaptively. Contrary to the current trend, in this work we hypothesize that a general inverse mapping from a random set of compressed measurements to the image domain exists for a given measurement basis, and can be learned. Such a model is single-shot, non-restrictive and does not parametrize the sampling process. To this end, we propose MOSAIC, a novel compressive sensing framework to reconstruct images given any random selection of measurements, sampled using a fixed basis. Motivated by the uneven distribution of information across measurements, MOSAIC incorporates an embedding technique to efficiently apply attention mechanisms on an encoded sequence of measurements, while dispensing the need to use unrolled deep networks. A range of experiments validate our proposed architecture as a promising alternative for existing CS reconstruction methods, by achieving the state-of-the-art for metrics of reconstruction accuracy on standard datasets.