# Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs

arXiv: https://arxiv.org/abs/2509.24640 · HTML: https://mlsanity.com/paper/2509.24640/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [Understanding Causality with Large Language Models: Feasibility and Opportunities](https://mlsanity.com/paper/2304.05524/) (2023)
- [TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models](https://mlsanity.com/paper/2311.17667/) (2023)
- [GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models](https://mlsanity.com/paper/2311.09048/) (2023)
- [iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs](https://mlsanity.com/paper/2502.03214/) (2025)
- [Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark](https://mlsanity.com/paper/2410.14702/) (2024)
- [M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models](https://mlsanity.com/paper/2405.15638/) (2024)
- [Learning to Compare Longitudinal Images](https://mlsanity.com/paper/2304.02531/) (2023)
- [Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs](https://mlsanity.com/paper/2407.00653/) (2024)
- [LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models](https://mlsanity.com/paper/2407.05434/) (2024)
- [Deep Multimodal Feature Encoding for Video Ordering](https://mlsanity.com/paper/2004.02205/) (2020)
- [MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning](https://mlsanity.com/paper/2306.02252/) (2023)

## Abstract (arXiv)

In this work, we introduce SPLICE, a human-curated benchmark derived from the COIN instructional video dataset, designed to probe event-based reasoning across multiple dimensions: temporal, causal, spatial, contextual, and general knowledge. SPLICE includes 3,381 human-filtered videos spanning 12 categories and 180 sub-categories, such as sports, engineering, and housework. These videos are segmented into a total of 11,423 event clips. We evaluate both human participants and state-of-the-art vision-language models (VLMs) on the task of rearranging these clips into coherent event sequences to assess visual reasoning capabilities. Results reveal a significant gap: VLMs struggle to match human performance. While human-annotated textual descriptions improve model accuracy, they do not affect human performance, suggesting that models rely more on language priors than on visual understanding. Even with annotations, VLMs fall short of human-level reasoning, underscoring persistent challenges in visual reasoning. A deeper analysis across sub-categories shows that VLMs perform relatively better on videos where temporal and causal reasoning are dominant, compared to those where contextual and spatial reasoning are dominant. They also perform better on everyday tasks than on specialized ones.