# Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration

arXiv: https://arxiv.org/abs/2306.06513 · HTML: https://mlsanity.com/paper/2306.06513/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 64 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 30 of 49)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Designing a Practical Degradation Model for Deep Blind Image Super-Resolution](https://mlsanity.com/paper/2103.14006/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Component Divide-and-Conquer for Real-World Image Super-Resolution](https://mlsanity.com/paper/2008.01928/) (2020)
- [Towards Robust Blind Face Restoration with Codebook Lookup Transformer](https://mlsanity.com/paper/2206.11253/) (2022)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [GAN Prior Embedded Network for Blind Face Restoration in the Wild](https://mlsanity.com/paper/2105.06070/) (2021)
- [Unsupervised Degradation Representation Learning for Blind Super-Resolution](https://mlsanity.com/paper/2104.00416/) (2021)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Learning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time](https://mlsanity.com/paper/2009.14468/) (2020)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)

## Built on by (most cited first, 14 of 14)

- [AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion](https://mlsanity.com/paper/2310.10123/) (2023)
- [Restoring Images in Adverse Weather Conditions via Histogram Transformer](https://mlsanity.com/paper/2407.10172/) (2024)
- [Learned Image Compression with Dictionary-based Entropy Model](https://mlsanity.com/paper/2504.00496/) (2025)
- [HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression](https://mlsanity.com/paper/2404.13372/) (2024)
- [TVC: Tokenized Video Compression with Ultra-Low Bit Rate](https://mlsanity.com/paper/2504.16953/) (2025)
- [Neural Image Compression Using Masked Sparse Visual Representation](https://mlsanity.com/paper/2309.11661/) (2023)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text](https://mlsanity.com/paper/2503.01261/) (2025)
- [Texture Vector-Quantization and Reconstruction Aware Prediction for Generative Super-Resolution](https://mlsanity.com/paper/2509.23774/) (2025)
- [GameIR: A Large-Scale Synthesized Ground-Truth Dataset for Image Restoration over Gaming Content](https://mlsanity.com/paper/2408.16866/) (2024)
- [mathttM3VIR: A Large-Scale Multi-Modality Multi-View Synthesized Benchmark Dataset for Image Restoration and Content Creation](https://mlsanity.com/paper/2509.16873/) (2025)
- [LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantization](https://mlsanity.com/paper/2601.00222/) (2026)
- [Enhancing Underwater Images via Adaptive Semantic-aware Codebook Learning](https://mlsanity.com/paper/2602.10586/) (2026)
- [Adaptive Fused Prior Transfer for Controllable Generative Image Compression](https://mlsanity.com/paper/2605.16817/) (2026)

## Abstract (arXiv)

Recent work on discrete generative priors, in the form of codebooks, has shown exciting performance for image reconstruction and restoration, as the discrete prior space spanned by the codebooks increases the robustness against diverse image degradations. Nevertheless, these methods require separate training of codebooks for different image categories, which limits their use to specific image categories only (e.g. face, architecture, etc.), and fail to handle arbitrary natural images. In this paper, we propose AdaCode for learning image-adaptive codebooks for class-agnostic image restoration. Instead of learning a single codebook for each image category, we learn a set of basis codebooks. For a given input image, AdaCode learns a weight map with which we compute a weighted combination of these basis codebooks for adaptive image restoration. Intuitively, AdaCode is a more flexible and expressive discrete generative prior than previous work. Experimental results demonstrate that AdaCode achieves state-of-the-art performance on image reconstruction and restoration tasks, including image super-resolution and inpainting.