# Human-In-The-Loop Document Layout Analysis

arXiv: https://arxiv.org/abs/2108.02095 · HTML: https://mlsanity.com/paper/2108.02095/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 74 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [Interventional Few-Shot Learning](https://mlsanity.com/paper/2009.13000/) (2020)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017)
- [Graph Few-shot Learning via Knowledge Transfer](https://mlsanity.com/paper/1910.03053/) (2019)
- [Interactive Policy Learning through Confidence-Based Autonomy](https://mlsanity.com/paper/1401.3439/) (2014)
- [Cross-Domain Document Object Detection: Benchmark Suite and Method](https://mlsanity.com/paper/2003.13197/) (2020)
- [Fully Convolutional Neural Networks for Page Segmentation of Historical Document Images](https://mlsanity.com/paper/1711.07695/) (2017)
- [Document Layout Analysis via Dynamic Residual Feature Fusion](https://mlsanity.com/paper/2104.02874/) (2021)

## Built on by (most cited first, 2 of 2)

- [Cross-Domain Document Layout Analysis Using Document Style Guide](https://mlsanity.com/paper/2201.09407/) (2022)
- [Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts](https://mlsanity.com/paper/2307.03003/) (2023)

## Abstract (arXiv)

Document layout analysis (DLA) aims to divide a document image into different types of regions. DLA plays an important role in the document content understanding and information extraction systems. Exploring a method that can use less data for effective training contributes to the development of DLA. We consider a Human-in-the-loop (HITL) collaborative intelligence in the DLA. Our approach was inspired by the fact that the HITL push the model to learn from the unknown problems by adding a small amount of data based on knowledge. The HITL select key samples by using confidence. However, using confidence to find key samples is not suitable for DLA tasks. We propose the Key Samples Selection (KSS) method to find key samples in high-level tasks (semantic segmentation) more accurately through agent collaboration, effectively reducing costs. Once selected, these key samples are passed to human beings for active labeling, then the model will be updated with the labeled samples. Hence, we revisited the learning system from reinforcement learning and designed a sample-based agent update strategy, which effectively improves the agent's ability to accept new samples. It achieves significant improvement results in two benchmarks (DSSE-200 (from 77.1% to 86.3%) and CS-150 (from 88.0% to 95.6%)) by using 10% of labeled data.