# Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents via Semantic-Oriented Hierarchical Graphs

arXiv: https://arxiv.org/abs/2305.01938 · HTML: https://mlsanity.com/paper/2305.01938/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 69 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [VisualMRC: Machine Reading Comprehension on Document Images](https://mlsanity.com/paper/2101.11272/) (2021)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [Hierarchical multimodal transformers for Multi-Page DocVQA](https://mlsanity.com/paper/2212.05935/) (2022)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://mlsanity.com/paper/2101.09465/) (2021)
- [StructuralLM: Structural Pre-training for Form Understanding](https://mlsanity.com/paper/2105.11210/) (2021)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022)

## Built on by (most cited first, 5 of 5)

- [SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding](https://mlsanity.com/paper/2411.01106/) (2024)
- [TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data](https://mlsanity.com/paper/2401.13223/) (2024)
- [WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts](https://mlsanity.com/paper/2506.15594/) (2025)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [Learning Deliberately, Acting Intuitively: Unlocking Test-Time Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2507.06999/) (2025)

## Abstract (arXiv)

Discrete reasoning over table-text documents (e.g., financial reports) gains increasing attention in recent two years. Existing works mostly simplify this challenge by manually selecting and transforming document pages to structured tables and paragraphs, hindering their practical application. In this work, we explore a more realistic problem setting in the form of TAT-DQA, i.e. to answer the question over a visually-rich table-text document. Specifically, we propose a novel Doc2SoarGraph framework with enhanced discrete reasoning capability by harnessing the differences and correlations among different elements (e.g., quantities, dates) of the given question and document with Semantic-oriented hierarchical Graph structures. We conduct extensive experiments on TAT-DQA dataset, and the results show that our proposed framework outperforms the best baseline model by 17.73% and 16.91% in terms of Exact Match (EM) and F1 score respectively on the test set, achieving the new state-of-the-art.