Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 88 references that are in the map. First posted on arXiv 17 Apr 2026.
Builds on
8 of 88 references.
Closest ancestors · 2
Same direction, newest first
- 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios27 Aug
- 2025Qwen3 Technical Report14 May
Other references · 6
- 2023GPT-4 Technical Reportcited by 18,132
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025OpenAI GPT-5 System Cardcited by 910
- 2025Kimi K2: Open Agentic Intelligencecited by 523
- 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 36
- 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models
- 2026SemBench: A Universal Semantic Framework for LLM Evaluation
- 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models
- 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
- 2024Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints
- 2026Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
- 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
- 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning