mlsanityData to 14 Oct 2026

Solution for SMART-101 Challenge of CVPR Multi-modal Algorithmic Reasoning Task 2024

arXiv 2406.05963 ↗First posted 10 Jun 2024cs.CV cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

In this paper, the solution of HYU MLLAB KT Team to the Multimodal Algorithmic Reasoning Task: SMART-101 CVPR 2024 Challenge is presented. Beyond conventional visual question-answering problems, the SMART-101 challenge aims to achieve human-level multimodal understanding by tackling complex visio-linguistic puzzles designed for children in the 6-8 age group. To solve this problem, we suggest two main ideas. First, to utilize the reasoning ability of a large-scale language model (LLM), the given visual cues (images) are grounded in the text modality. For this purpose, we generate highly detailed text captions that describe the context of the image and use these captions as input for the LLM. Second, due to the nature of puzzle images, which often contain various geometric visual patterns, we utilize an object detection algorithm to ensure these patterns are not overlooked in the captioning process. We employed the SAM algorithm, which can detect various-size objects, to capture the visual features of these geometric patterns and used this information as input for the LLM. Under the puzzle split configuration, we achieved an option selection accuracy Oacc of 29.5 on the test set and a weighted option selection accuracy (WOSA) of 27.1 on the challenge set.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 11 of its 27 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

11 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Segment Anythingcited by 8,234
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  5. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  6. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  7. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  8. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning
  2. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
  3. 2024Smart Vision-Language Reasoners
  4. 2025Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions
  5. 2026Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education
  6. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  8. 2024HAMMR: HierArchical MultiModal React agents for generic VQA