Towards a Unified Multimodal Reasoning Framework
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Recent advancements in deep learning have led to the development of powerful language models (LMs) that excel in various tasks. Despite these achievements, there is still room for improvement, particularly in enhancing reasoning abilities and incorporating multimodal data. This report investigates the potential impact of combining Chain-of-Thought (CoT) reasoning and Visual Question Answering (VQA) techniques to improve LM's accuracy in solving multiple-choice questions. By employing TextVQA and ScienceQA datasets, we assessed the effectiveness of three text embedding methods and three visual embedding approaches. Our experiments aimed to fill the gap in current research by investigating the combined impact of CoT and VQA, contributing to the understanding of how these techniques can improve the reasoning capabilities of state-of-the-art models like GPT-4. Results from our experiments demonstrated the potential of these approaches in enhancing LM's reasoning and question-answering capabilities, providing insights for further research and development in the field, and paving the way for more accurate and reliable AI systems that can handle complex reasoning tasks across multiple modalities.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 23 references that are in the map. First posted on arXiv 22 Dec 2023.
Builds on
16 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2019Towards VQA Models That Can Readcited by 1,696
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
- 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
- 2023InstructABSA: Instruction Learning for Aspect Based Sentiment Analysiscited by 22
- 2023Instruction Tuned Models are Quick Learnerscited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
- 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
- 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios
- 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework
- 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
- 2026TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding