# DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading

arXiv: https://arxiv.org/abs/2310.14802 · HTML: https://mlsanity.com/paper/2310.14802/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding](https://mlsanity.com/paper/2203.06947/) (2022)
- [Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solution](https://mlsanity.com/paper/2102.06732/) (2021)
- [Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention](https://mlsanity.com/paper/2010.07891/) (2020)
- [StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training](https://mlsanity.com/paper/2303.00289/) (2023)
- [Entity Recognition at First Sight: Improving NER with Eye Movement Information](https://mlsanity.com/paper/1902.10068/) (2019)
- [ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction](https://mlsanity.com/paper/2106.10786/) (2021)
- [PP-StructureV2: A Stronger Document Analysis System](https://mlsanity.com/paper/2210.05391/) (2022)
- [Leveraging Cognitive Features for Sentiment Analysis](https://mlsanity.com/paper/1701.05581/) (2017)
- [Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze](https://mlsanity.com/paper/2011.04592/) (2020)
- [CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals](https://mlsanity.com/paper/2106.05544/) (2021)
- [Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks](https://mlsanity.com/paper/2001.05152/) (2020)

## Built on by (most cited first, 2 of 2)

- [Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding](https://mlsanity.com/paper/2409.19672/) (2024)
- [Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documents](https://mlsanity.com/paper/2403.15765/) (2024)

## Abstract (arXiv)

The use of visually-rich documents (VRDs) in various fields has created a demand for Document AI models that can read and comprehend documents like humans, which requires the overcoming of technical, linguistic, and cognitive barriers. Unfortunately, the lack of appropriate datasets has significantly hindered advancements in the field. To address this issue, we introduce textscDocTrack, a VRD dataset really aligned with human eye-movement information using eye-tracking technology. This dataset can be used to investigate the challenges mentioned above. Additionally, we explore the impact of human reading order on document understanding tasks and examine what would happen if a machine reads in the same order as a human. Our results suggest that although Document AI models have made significant progress, they still have a long way to go before they can read VRDs as accurately, continuously, and flexibly as humans do. These findings have potential implications for future research and development of Document AI models. The data is available at urlhttps://github.com/hint-lab/doctrack.