# Learning Multiple Representations with Inconsistency-Guided Detail Regularization for Mask-Guided Matting

arXiv: https://arxiv.org/abs/2403.19213 · HTML: https://mlsanity.com/paper/2403.19213/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Semantic Flow for Fast and Accurate Scene Parsing](https://mlsanity.com/paper/2002.10120/) (2020)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Towards High-Resolution Salient Object Detection](https://mlsanity.com/paper/1908.07274/) (2019)
- [Bridging Composite and Real: Towards End-to-end Deep Image Matting](https://mlsanity.com/paper/2010.16188/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation](https://mlsanity.com/paper/1909.09725/) (2019)
- [Natural Image Matting via Guided Contextual Attention](https://mlsanity.com/paper/2001.04069/) (2020)
- [End-to-End Wireframe Parsing](https://mlsanity.com/paper/1905.03246/) (2019)
- [Learning to Parse Wireframes in Images of Man-Made Environments](https://mlsanity.com/paper/2007.07527/) (2020)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Devil in the Details: Towards Accurate Single and Multiple Human Parsing](https://mlsanity.com/paper/1809.05996/) (2018)
- [Pyramid Grafting Network for One-Stage High Resolution Saliency Detection](https://mlsanity.com/paper/2204.05041/) (2022)
- [Privacy-Preserving Portrait Matting](https://mlsanity.com/paper/2104.14222/) (2021)
- [DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients](https://mlsanity.com/paper/2212.07766/) (2022)
- [Deep Automatic Natural Image Matting](https://mlsanity.com/paper/2107.07235/) (2021)
- [Semantic Image Matting](https://mlsanity.com/paper/2104.08201/) (2021)
- [Learning Attraction Field Representation for Robust Line Segment Detection](https://mlsanity.com/paper/1812.02122/) (2018)
- [AlphaGAN: Generative adversarial networks for natural image matting](https://mlsanity.com/paper/1807.10088/) (2018)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [MatteFormer: Transformer-Based Image Matting via Prior-Tokens](https://mlsanity.com/paper/2203.15662/) (2022)
- [Disentangled Image Matting](https://mlsanity.com/paper/1909.04686/) (2019)
- [PPGNet: Learning Point-Pair Graph for Line Segment Detection](https://mlsanity.com/paper/1905.03415/) (2019)
- [Learning Affinity-Aware Upsampling for Deep Image Matting](https://mlsanity.com/paper/2011.14288/) (2020)

## Abstract (arXiv)

Mask-guided matting networks have achieved significant improvements and have shown great potential in practical applications in recent years. However, simply learning matting representation from synthetic and lack-of-real-world-diversity matting data, these approaches tend to overfit low-level details in wrong regions, lack generalization to objects with complex structures and real-world scenes such as shadows, as well as suffer from interference of background lines or textures. To address these challenges, in this paper, we propose a novel auxiliary learning framework for mask-guided matting models, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations. Our framework and model introduce the following key aspects: (1) to learn real-world adaptive semantic representation for objects with diverse and complex structures under real-world scenes, we introduce extra semantic segmentation and edge detection tasks on more diverse real-world data with segmentation annotations; (2) to avoid overfitting on low-level details, we propose a module to utilize the inconsistency between learned segmentation and matting representations to regularize detail refinement; (3) we propose a novel background line detection task into our auxiliary learning framework, to suppress interference of background lines or textures. In addition, we propose a high-quality matting benchmark, Plant-Mat, to evaluate matting methods on complex structures. Extensively quantitative and qualitative results show that our approach outperforms state-of-the-art mask-guided methods.