mlsanityData to 14 Oct 2026

Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA

arXiv 1911.06258 ↗First posted 14 Nov 2019cs.CV cs.CLcited by 107

Not on the map. Why?

Abstract (arXiv)

Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, we propose a novel model for the TextVQA task based on a multimodal transformer architecture accompanied by a rich representation for text in images. Our model naturally fuses different modalities homogeneously by embedding them into a common semantic space where self-attention is applied to model inter- and intra- modality context. Furthermore, it enables iterative answer decoding with a dynamic pointer network, allowing the model to form an answer through multi-step prediction instead of one-step classification. Our model outperforms existing approaches on three benchmark datasets for the TextVQA task by a large margin.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 26; 2023: 23; 2024: 14; 2025: 9; 2026: 6.

107 papers.

Sort:

By area:

  1. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  3. 2021InfographicVQAcited by 437
  4. 2021A Survey of Transformerscited by 401
  5. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  6. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  7. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  8. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  9. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  10. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  11. 2021M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformerscited by 74
  12. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
38 more
  1. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  2. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  3. 2021Human-Adversarial Visual Question Answeringcited by 57
  4. 2022MulT: An End-to-End Multitask Learning Transformercited by 48
  5. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  6. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  7. 2021Split, embed and merge: An accurate table structure recognizercited by 42
  8. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  9. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  10. 2021Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Featurescited by 34
  11. 2022Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrievalcited by 34
  12. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  13. 2023PDFVQA: A New Dataset for Real-World VQA on PDF Documentscited by 28
  14. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  15. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  16. 2021Towards Accurate Text-based Image Captioning with Content Diversity Explorationcited by 25
  17. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 25
  18. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  19. 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
  20. 2023ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embeddingcited by 23
  21. 2021Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsulescited by 22
  22. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  23. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 21
  24. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  25. 2023OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamesecited by 20
  26. 2024LoFormer: Local Frequency Transformer for Image Deblurringcited by 20
  27. 2021ICDAR 2021 Competition on Document VisualQuestion Answeringcited by 19
  28. 2021PAM: Understanding Product Images in Cross Product Category Attribute Extractioncited by 18
  29. 2023DocTr: Document Transformer for Structured Information Extraction in Documentscited by 18
  30. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  31. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  32. 2022Controllable Image Captioning via Promptingcited by 15
  33. 2023Towards Models that Can See and Readcited by 15
  34. 2021A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanationscited by 14
  35. 2021Localize, Group, and Select: Boosting Text-VQA by Scene Text Modelingcited by 14
  36. 2022COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Textscited by 13
  37. 2022TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generationcited by 12
  38. 2023Learning Unseen Modality Interactioncited by 12

50 most cited of 107 All 107 (JSON)