# Attention based Occlusion Removal for Hybrid Telepresence Systems

arXiv: https://arxiv.org/abs/2112.01098 · HTML: https://mlsanity.com/paper/2112.01098/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [3FabRec: Fast Few-shot Face alignment by Reconstruction](https://mlsanity.com/paper/1911.10448/) (2019)
- [FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality](https://mlsanity.com/paper/1610.03151/) (2016)
- [LaFIn: Generative Landmark Guided Face Inpainting](https://mlsanity.com/paper/1911.11394/) (2019)
- [Egocentric Videoconferencing](https://mlsanity.com/paper/2107.03109/) (2021)
- [AFN: Attentional Feedback Network based 3D Terrain Super-Resolution](https://mlsanity.com/paper/2010.01626/) (2020)

## Built on by (most cited first, 2 of 2)

- [Facial De-occlusion Network for Virtual Telepresence Systems](https://mlsanity.com/paper/2210.12622/) (2022)
- [Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XR](https://mlsanity.com/paper/2508.12336/) (2025)

## Abstract (arXiv)

Traditionally, video conferencing is a widely adopted solution for telecommunication, but a lack of immersiveness comes inherently due to the 2D nature of facial representation. The integration of Virtual Reality (VR) in a communication/telepresence system through Head Mounted Displays (HMDs) promises to provide users a much better immersive experience. However, HMDs cause hindrance by blocking the facial appearance and expressions of the user. To overcome these issues, we propose a novel attention-enabled encoder-decoder architecture for HMD de-occlusion. We also propose to train our person-specific model using short videos (1-2 minutes) of the user, captured in varying appearances, and demonstrated generalization to unseen poses and appearances of the user. We report superior qualitative and quantitative results over state-of-the-art methods. We also present applications of this approach to hybrid video teleconferencing using existing animation and 3D face reconstruction pipelines.