# The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping

arXiv: https://arxiv.org/abs/2510.08482 · HTML: https://mlsanity.com/paper/2510.08482/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 11 of 11)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [MiMo-VL Technical Report](https://mlsanity.com/paper/2506.03569/) (2025)
- [Metaphor Understanding Challenge Dataset for LLMs](https://mlsanity.com/paper/2403.11810/) (2024)
- [Gesture-Informed Robot Assistance via Foundation Models](https://mlsanity.com/paper/2309.02721/) (2023)
- [Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.16427/) (2025)
- [Learning to Generalize without Bias for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2502.20158/) (2025)
- [SignLLM: Sign Language Production Large Language Models](https://mlsanity.com/paper/2405.10718/) (2024)

## Abstract (arXiv)

Iconicity, the resemblance between linguistic form and meaning, is pervasive in signed languages, offering a natural testbed for visual grounding. For vision-language models (VLMs), the challenge is to recover such essential mappings from dynamic human motion rather than static context. We introduce the Visual Iconicity Challenge, a novel video-based benchmark that adapts psycholinguistic measures to evaluate VLMs on three tasks: (i) phonological sign-form prediction (e.g., handshape, location), (ii) transparency (inferring meaning from visual form), and (iii) graded iconicity ratings. We assess 13 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands and compare them to human baselines. On phonological form prediction, VLMs recover some handshape and location detail but remain below human performance; on transparency, they are far from human baselines; and only top models correlate moderately with human iconicity ratings. Interestingly, models with stronger phonological form prediction correlate better with human iconicity judgment, indicating shared sensitivity to visually grounded structure. Our findings validate these diagnostic tasks and motivate human-centric signals and embodied learning methods for modelling iconicity and improving visual grounding in multimodal models.