mlsanityData to 7 Oct 2026

Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering

arXiv 2204.10448 ↗First posted 22 Apr 2022cs.CV cs.AI cs.CLcited by 15

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Knowledge-based visual question answering (QA) aims to answer a question which requires visually-grounded external knowledge beyond image content itself. Answering complex questions that require multi-hop reasoning under weak supervision is considered as a challenging problem since i) no supervision is given to the reasoning process and ii) high-order semantics of multi-hop knowledge facts need to be captured. In this paper, we introduce a concept of hypergraph to encode high-level semantics of a question and a knowledge base, and to learn high-order associations between them. The proposed model, Hypergraph Transformer, constructs a question hypergraph and a query-aware knowledge hypergraph, and infers an answer by encoding inter-associations between two hypergraphs and intra-associations in both hypergraph itself. Extensive experiments on two knowledge-based visual QA and two knowledge-based textual QA demonstrate the effectiveness of our method, especially for multi-hop reasoning problem. Our source code is available at https://github.com/yujungheo/kbvqa-public.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 52 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

25 of 52 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  8. 2015Gated Graph Sequence Neural Networkscited by 792
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2014Memory Networkscited by 354
  12. 2015Visual7W: Grounded Question Answering in Imagescited by 315
13 more
  1. 2018Bilinear Attention Networkscited by 277
  2. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  3. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  4. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  5. 2017Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learningcited by 161
  6. 2019Knowledge Graph Alignment Network with Gated Multi-hop Neighborhood Aggregationcited by 101
  7. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  8. 2020Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graphcited by 57
  9. 2014Question Answering with Subgraph Embeddingscited by 56
  10. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  11. 2018An Interpretable Reasoning Network for Multi-Relation Question Answeringcited by 47
  12. 2018Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answeringcited by 30
  13. 2014Open Question Answering with Weakly Supervised Embedding Modelscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 3; 2025: 2; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  2. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  3. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  4. 2023Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modalscited by 24
  5. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  6. 2022KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understandingcited by 9
  7. 2023Hypergraph Transformer for Semi-Supervised Classificationcited by 7
  8. 2022Dense-ATOMIC: Towards Densely-connected ATOMIC with High Knowledge Coverage and Massive Multi-hop Pathscited by 6
  9. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5
  10. 2026Higher-Order Knowledge Representations for Agentic Scientific Reasoningcited by 3
  11. 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answeringcited by 1
  12. 2024Hierarchical Multi-modal Transformer for Cross-modal Long Document Classificationcited by 0
3 more
  1. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  2. 2026SoftReason: A Fully Differentiable Neuro-Soft-Symbolic Deductive Reasoning Architecture over High-Dimensional Perceptual Datacited by 0
  3. 2026HyperLabel: Multi-Label Classification via Hypergraph-Based Label Correlation Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering
  2. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  3. 2022From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering
  4. 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
  5. 2024Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA
  6. 2026HyperProve: Answer-Guided Hypergraph Expansion for Multi-Hop Question Answering
  7. 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering
  8. 2025Multi-Hop Reasoning for Question Answering with Hyperbolic Representations