mlsanityData to 7 Oct 2026

Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modals

arXiv 2308.06207 ↗First posted 11 Aug 2023cs.CLcited by 24

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Reasoning ability is one of the most crucial capabilities of a foundation model, signifying its capacity to address complex reasoning tasks. Chain-of-Thought (CoT) technique is widely regarded as one of the effective methods for enhancing the reasoning ability of foundation models and has garnered significant attention. However, the reasoning process of CoT is linear, step-by-step, similar to personal logical reasoning, suitable for solving general and slightly complicated problems. On the contrary, the thinking pattern of an expert owns two prominent characteristics that cannot be handled appropriately in CoT, i.e., high-order multi-hop reasoning and multimodal comparative judgement. Therefore, the core motivation of this paper is transcending CoT to construct a reasoning paradigm that can think like an expert. The hyperedge of a hypergraph could connect various vertices, making it naturally suitable for modelling high-order relationships. Inspired by this, this paper innovatively proposes a multimodal Hypergraph-of-Thought (HoT) reasoning paradigm, which enables the foundation models to possess the expert-level ability of high-order multi-hop reasoning and multimodal comparative judgement. Specifically, a textual hypergraph-of-thought is constructed utilizing triple as the primary thought to model higher-order relationships, and a hyperedge-of-thought is generated through multi-hop walking paths to achieve multi-hop inference. Furthermore, we devise a visual hypergraph-of-thought to interact with the textual hypergraph-of-thought via Cross-modal Co-Attention Graph Learning for multimodal comparative verification. Experimentations on the ScienceQA benchmark demonstrate the proposed HoT-based T5 outperforms CoT-based GPT3.5 and chatGPT, which is on par with CoT-based GPT4 with a lower model size.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 42 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

36 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step24 Jun
  2. 2022Large Language Models Are Reasoning Teachers20 Dec
  3. 2022Teaching Small Language Models to Reason16 Dec
  4. 2022Large Language Models are Zero-Shot Reasoners24 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
18 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023BloombergGPT: A Large Language Model for Financecited by 923
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  9. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  10. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  11. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  12. 2018Bilinear Attention Networkscited by 277
  13. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  14. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  15. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  16. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  17. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  18. 2022Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answeringcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 10; 2026: 3.

24 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  6. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  8. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  9. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  10. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  11. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  12. 2024The Role of Deductive and Inductive Reasoning in Large Language Modelscited by 20
12 more
  1. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  2. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  3. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  4. 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmarkcited by 10
  5. 2023Towards A Unified View of Answer Calibration for Multi-Step Reasoningcited by 8
  6. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  7. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  8. 2026Higher-Order Knowledge Representations for Agentic Scientific Reasoningcited by 3
  9. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  10. 2024T2 of Thoughts: Temperature Tree Elicits Reasoning in Large Language Modelscited by 1
  11. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  12. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought
  2. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
  3. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
  4. 2025Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
  5. 2025RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoning
  6. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
  7. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  8. 2024A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning