GEB-Bench: Abstract Structures Told in Many Voices
Not on the map. Why?
Abstract (arXiv)
Analogy allows models to apply relational knowledge from one situation to another. Benchmarks of equivalent representations test different ways of expressing the same underlying problem, whereas analogy often links situations involving different objects and content. To evaluate the analogy ability of models, we introduce GEB-Bench, a benchmark centered on cross-voice motif matching. A voice presents a motif as a scene, story, mathematical statement, or programmatic skeleton. The first task is identification across these voices. The second is matching scenes to stories or mathematical statements, and stories to mathematical statements, by their shared motif. A complete construction pipeline generates items in these voices from the relation specified for each motif, checks the order and repetition of story units, and assembles paired questions. We evaluate current models on our benchmark. Results show that matching performance depends on the target voice. The paired scene-to-story study reveals that (1) evaluated models still make matching errors when they separately identify the source and every candidate correctly, and (2) identification improves more consistently than scene-to-story matching across the evaluated Qwen model sizes. These findings show that successful motif identification does not ensure reliable matching across voices.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
21 of 33 references.
References · 21
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2021Knowledge Neurons in Pretrained Transformerscited by 601
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
- 2023Function Vectors in Large Language Modelscited by 308
- 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
- 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
9 more
- 2018Measuring abstract reasoning in neural networkscited by 166
- 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
- 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
- 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
- 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
- 2024Knowledge Circuits in Pretrained Transformerscited by 60
- 2019Learning to Make Analogies by Contrasting Abstract Relational Structurecited by 53
- 2019V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matricescited by 24
- 2025SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Modelscited by 6
Built on by
Not cited yet.