mlsanityData to 7 Oct 2026

OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition

arXiv 2403.19128 ↗First posted 28 Mar 2024cs.CVcited by 74

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, visually-situated text parsing (VsTP) has experienced notable advancements, driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions. Various methods have been proposed to address the challenging problem of VsTP. However, due to the diversified targets and heterogeneous schemas, previous works usually design task-specific architectures and objectives for individual tasks, which inadvertently leads to modal isolation and complex workflow. In this paper, we propose a unified paradigm for parsing visually-situated text across diverse scenarios. Specifically, we devise a universal model, called OmniParser, which can simultaneously handle three typical visually-situated text parsing tasks: text spotting, key information extraction, and table recognition. In OmniParser, all tasks share the unified encoder-decoder architecture, the unified objective: point-conditioned text generation, and the unified input & output representation: prompt & structured sequences. Extensive experiments demonstrate that the proposed OmniParser achieves state-of-the-art (SOTA) or highly competitive performances on 7 datasets for the three visually-situated text parsing tasks, despite its unified, concise design. The code is available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 82 of its 99 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

82 of 99 references.

References · 82

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  5. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  7. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  8. 2021OCR-free Document Understanding Transformercited by 400
  9. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  10. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  11. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  12. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
38 more
  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  2. 2017EAST: An Efficient and Accurate Scene Text Detectorcited by 248
  3. 2019Image-based table recognition: data, model, and evaluationcited by 233
  4. 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
  5. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  6. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  7. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  8. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  9. 2020Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spottingcited by 114
  10. 2018FOTS: Fast Oriented Text Spotting with a Unified Networkcited by 109
  11. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  12. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  13. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  14. 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
  15. 2022TableFormer: Table Structure Understanding with Transformerscited by 93
  16. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  17. 2022Unified Pretraining Framework for Document Understandingcited by 85
  18. 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
  19. 2016Image-to-Markup Generation with Coarse-to-Fine Attentioncited by 79
  20. 2021LayoutReader: Pre-training of Text and Layout for Reading Order Detectioncited by 79
  21. 2022DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spottingcited by 74
  22. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  23. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  24. 2020PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networkscited by 70
  25. 2022SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognitioncited by 70
  26. 2022Text Spotting Transformerscited by 70
  27. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  28. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  29. 2019All You Need Is Boundary: Toward Arbitrary-Shaped Text Spottingcited by 66
  30. 2023SPTS v2: Single-Point Scene Text Spottingcited by 66
  31. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  32. 2021PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Textcited by 65
  33. 2021Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solutioncited by 64
  34. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  35. 2019Convolutional Character Networkscited by 61
  36. 2020TRIE: End-to-End Text Reading and Information Extraction for Document Understandingcited by 61
  37. 2019Towards Unconstrained End-to-End Text Spottingcited by 57
  38. 2017Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networkscited by 56

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 47; 2026: 15.

74 papers.

Sort:

By area:

  1. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  2. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  3. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  4. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  5. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  6. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  8. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 44
  9. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  10. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  11. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  12. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
38 more
  1. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  2. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  3. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  4. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  5. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  6. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  7. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  8. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  9. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 14
  10. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  11. 2025Mano Technical Reportcited by 13
  12. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  13. 2024Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasonercited by 9
  14. 2024DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthinesscited by 8
  15. 2024InstructOCR: Instruction Boosting Scene Text Spottingcited by 7
  16. 2025Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Modelscited by 7
  17. 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 6
  18. 2025TABLET: Table Structure Recognition using Encoder-only Transformerscited by 5
  19. 2024VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizercited by 4
  20. 2025DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learningcited by 4
  21. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  22. 2025Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automationcited by 4
  23. 2025UI-UG: A Unified MLLM for UI Understanding and Generationcited by 4
  24. 2025AutoPK: Leveraging LLMs and a Hybrid Similarity Metric for Advanced Retrieval of Pharmacokinetic Data from Complex Tables and Documentscited by 4
  25. 2025LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spottingcited by 4
  26. 2025Exploring LLMs for Scientific Information Extraction Using The SciEx Frameworkcited by 4
  27. 2024MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysiscited by 3
  28. 2024See then Tell: Enhancing Key Information Extraction with Vision Groundingcited by 3
  29. 2025Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automationcited by 3
  30. 2025PixelWeb: The First Web GUI Dataset with Pixel-Wise Labelscited by 3
  31. 2025Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agentcited by 3
  32. 2025Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learningcited by 3
  33. 2025From Fragment to One Piece: A Survey on AI-Driven Graphic Designcited by 2
  34. 2025PILOT: A Promptable Interleaved Layout-aware OCR Transformercited by 2
  35. 2025DocSpiral: A Platform for Integrated Assistive Document Annotation through Human-in-the-Spiralcited by 2
  36. 2025Efficient Medical VIE via Reinforcement Learningcited by 2
  37. 2025Class-Agnostic Region-of-Interest Matching in Document Imagescited by 2
  38. 2025Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agentscited by 2

50 most cited of 74 All 74 (JSON)