mlsanityData to 7 Oct 2026

Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

arXiv 2102.09550 ↗First posted 18 Feb 2021cs.CL cs.LGcited by 134

Not on the map. Why?

Abstract (arXiv)

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

48 of 63 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
36 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  2. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  5. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  8. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  9. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  10. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  13. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  14. 2018QuAC : Question Answering in Contextcited by 379
  15. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  16. 2019Axial Attention in Multidimensional Transformerscited by 350
  17. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  18. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  19. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  20. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  21. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  22. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  23. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116
  24. 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
  25. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  26. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63
  27. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  28. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  29. 2017CloudScan - A configuration-free invoice analysis system using recurrent neural networkscited by 33
  30. 2020A Study of Non-autoregressive Model for Sequence Generationcited by 31
  31. 2020Kleister: A novel task for Information Extraction involving Long Documents with Complex Layoutcited by 27
  32. 2016WikiReading: A Novel Large-scale Language Understanding Task over Wikipediacited by 26
  33. 2018Adversarial TableQA: Attention Supervision for Question Answering on Tablescited by 10
  34. 2019Unifying Question Answering, Text Classification, and Regression via Span Extractioncited by 8
  35. 2020From Dataset Recycling to Multi-Property Extraction and Beyondcited by 6
  36. 2020UniCase -- Rethinking Casing in Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 28; 2023: 44; 2024: 30; 2025: 21; 2026: 3.

134 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  2. 2021InfographicVQAcited by 430
  3. 2021OCR-free Document Understanding Transformercited by 400
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  6. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  7. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  8. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  9. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  10. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  11. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  12. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
38 more
  1. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  2. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  3. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  4. 2022Unified Pretraining Framework for Document Understandingcited by 85
  5. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  6. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  7. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  8. 2022DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognitioncited by 62
  9. 2021Document AI: Benchmarks, Models and Applicationscited by 61
  10. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  11. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  12. 2021MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understandingcited by 44
  13. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  14. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  15. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  16. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  17. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  18. 2022WebFormer: The Web-page Transformer for Structure Information Extractioncited by 34
  19. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  20. 2024LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understandingcited by 24
  21. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  22. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  23. 2023CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Datacited by 21
  24. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  25. 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understandingcited by 20
  26. 2023FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extractioncited by 20
  27. 2021ICDAR 2021 Competition on Document VisualQuestion Answeringcited by 19
  28. 2023DocParser: End-to-end OCR-free Information Extraction from Visually Rich Documentscited by 19
  29. 2022Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understandingcited by 18
  30. 2023Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentrationcited by 18
  31. 2022Doc2Graph: a Task Agnostic Document Understanding Framework based on Graph Neural Networkscited by 17
  32. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17
  33. 2023Privacy-Aware Document Visual Question Answeringcited by 17
  34. 2022Evaluating Out-of-Distribution Performance on Document Image Classifierscited by 16
  35. 2022STable: Table Generation Framework for Encoder-Decoder Modelscited by 15
  36. 2022VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classificationcited by 14
  37. 2024Improving Language Understanding from Screenshotscited by 14
  38. 2023SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentationcited by 13

50 most cited of 134 All 134 (JSON)