# SAIR: Learning Semantic-aware Implicit Representation

arXiv: https://arxiv.org/abs/2310.09285 · HTML: https://mlsanity.com/paper/2310.09285/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [Local Texture Estimator for Implicit Representation Function](https://mlsanity.com/paper/2111.08918/) (2021)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [S-NeRF: Neural Radiance Fields for Street Views](https://mlsanity.com/paper/2303.00749/) (2023)
- [Recurrent Feature Reasoning for Image Inpainting](https://mlsanity.com/paper/2008.03737/) (2020)
- [TAP: Text-Aware Pre-training for Text-VQA and Text-Caption](https://mlsanity.com/paper/2012.04638/) (2020)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting](https://mlsanity.com/paper/2203.06304/) (2022)
- [DISCO: Adversarial Defense with Local Implicit Functions](https://mlsanity.com/paper/2212.05630/) (2022)
- [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2203.15799/) (2022)
- [Generalised Implicit Neural Representations](https://mlsanity.com/paper/2205.15674/) (2022)
- [Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations](https://mlsanity.com/paper/2012.01644/) (2020)

## Built on by (most cited first, 3 of 3)

- [Bidirectional Autoregressive Diffusion Model for Dance Generation](https://mlsanity.com/paper/2402.04356/) (2024)
- [DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2505.11676/) (2025)
- [CLII: Visual-Text Inpainting via Cross-Modal Predictive Interaction](https://mlsanity.com/paper/2407.16204/) (2024)

## Abstract (arXiv)

Implicit representation of an image can map arbitrary coordinates in the continuous domain to their corresponding color values, presenting a powerful capability for image reconstruction. Nevertheless, existing implicit representation approaches only focus on building continuous appearance mapping, ignoring the continuities of the semantic information across pixels. As a result, they can hardly achieve desired reconstruction results when the semantic information within input images is corrupted, for example, a large region misses. To address the issue, we propose to learn semantic-aware implicit representation (SAIR), that is, we make the implicit representation of each pixel rely on both its appearance and semantic information (eg, which object does the pixel belong to). To this end, we propose a framework with two modules: (1) building a semantic implicit representation (SIR) for a corrupted image whose large regions miss. Given an arbitrary coordinate in the continuous domain, we can obtain its respective text-aligned embedding indicating the object the pixel belongs. (2) building an appearance implicit representation (AIR) based on the SIR. Given an arbitrary coordinate in the continuous domain, we can reconstruct its color whether or not the pixel is missed in the input. We validate the novel semantic-aware implicit representation method on the image inpainting task, and the extensive experiments demonstrate that our method surpasses state-of-the-art approaches by a significant margin.