# Multi-task Planar Reconstruction with Feature Warping Guidance

arXiv: https://arxiv.org/abs/2311.14981 · HTML: https://mlsanity.com/paper/2311.14981/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 25 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers](https://mlsanity.com/paper/2111.14600/) (2021)
- [You Only Look One-level Feature](https://mlsanity.com/paper/2103.09460/) (2021)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)
- [PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1804.06278/) (2018)
- [Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding](https://mlsanity.com/paper/1902.09777/) (2019)
- [PlaneTR: Structure-Guided Transformers for 3D Plane Recovery](https://mlsanity.com/paper/2107.13108/) (2021)
- [PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2203.12082/) (2022)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [Planckian Jitter: countering the color-crippling effects of color jitter on self-supervised training](https://mlsanity.com/paper/2202.07993/) (2022)
- [PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2110.11219/) (2021)
- [PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN](https://mlsanity.com/paper/2103.15428/) (2021)
- [Learning to Correct 3D Reconstructions from Multiple Views](https://mlsanity.com/paper/2001.08098/) (2020)

## Abstract (arXiv)

Piece-wise planar 3D reconstruction simultaneously segments plane instances and recovers their 3D plane parameters from an image, which is particularly useful for indoor or man-made environments. Efficient reconstruction of 3D planes coupled with semantic predictions offers advantages for a wide range of applications requiring scene understanding and concurrent spatial mapping. However, most existing planar reconstruction models either neglect semantic predictions or do not run efficiently enough for real-time applications. We introduce SOLOPlanes, a real-time planar reconstruction model based on a modified instance segmentation architecture which simultaneously predicts semantics for each plane instance, along with plane parameters and piece-wise plane instance masks. We achieve an improvement in instance mask segmentation by including multi-view guidance for plane predictions in the training process. This cross-task improvement, training for plane prediction but improving the mask segmentation, is due to the nature of feature sharing in multi-task learning. Our model simultaneously predicts semantics using single images at inference time, while achieving real-time predictions at 43 FPS.