mlsanityData to 7 Oct 2026

ProTo: Program-Guided Transformer for Program-Guided Tasks

arXiv 2110.00804 ↗First posted 2 Oct 2021cs.LG cs.AI cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision making, we formulate program-guided tasks which require learning to execute a given program on the observed task specification. Furthermore, we propose the Program-guided Transformer (ProTo), which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-attention to pass messages between the specification and routines in the program. ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches. We demonstrate that ProTo significantly outperforms the previous state-of-the-art methods on GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally, ProTo demonstrates better generalization to unseen, complex, and human-written programs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 64 of its 107 references that are in the map. First posted on arXiv 2 Oct 2021.

Builds on

64 of 107 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2017Graph Attention Networkscited by 8,854
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2014Neural Turing Machinescited by 630
  8. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 389
  9. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  10. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  13. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  14. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  15. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  16. 2017Neural-Symbolic Learning and Reasoning: A Survey and Interpretationcited by 195
  17. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  18. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  19. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  20. 2019Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoningcited by 149
  21. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  22. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  23. 2015Neural GPUs Learn Algorithmscited by 118
  24. 2020One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Controlcited by 114
  25. 2019Learning by Abstraction: The Neural State Machinecited by 113
  26. 2015Neural Programmer-Interpreterscited by 96
  27. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  28. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  29. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  30. 2021ResT: An Efficient Transformer for Visual Recognitioncited by 79
  31. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  32. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  33. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64
  34. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  35. 2018Explainable Neural Computation via Stack Neural Module Networkscited by 54
  36. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  37. 2019Meta Module Network for Compositional Visual Reasoningcited by 39
  38. 2021Container: Context Aggregation Networkcited by 37

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 5; 2024: 1; 2025: 1; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  4. 2022Bridging the Domain Gap for Multi-Agent Perceptioncited by 60
  5. 2022Tracking Objects as Pixel-wise Distributionscited by 36
  6. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  7. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  8. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  9. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  10. 2023Grounded Image Text Matching with Mismatched Relation Reasoningcited by 8
  11. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  12. 2022E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidancecited by 6
7 more
  1. 2022LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-trainingcited by 5
  2. 2022End-to-end View Synthesis via NeRF Attentioncited by 5
  3. 2022Visually Grounded VQA by Lattice-based Retrievalcited by 3
  4. 2023Detection-based Intermediate Supervision for Visual Question Answeringcited by 3
  5. 2022Towards Reasoning-Aware Explainable VQAcited by 2
  6. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  7. 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0