BaziQA-Benchmark: Evaluating Symbolic and Temporally Compositional Reasoning in Large Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We present BaziQA-Benchmark, a standardized benchmark for evaluating symbolic and temporally compositional reasoning in large language models. The benchmark is derived from 200 professionally curated, multiple-choice problems from the Global Fortune-teller Competition (2021--2025), where each instance requires structured inference over a fixed symbolic chart and interacting temporal conditions. Unlike anecdotal or prompt-driven evaluations, BaziQA-Benchmark enables objective scoring and controlled comparison across years, domains, and model families. We evaluate contemporary language models under a multi-turn setting and analyze performance variation across temporal difficulty, reasoning domains, and inference protocols.To further probe reasoning behavior, we introduce a lightweight Structured Reasoning Protocol that constrains inference order without adding domain knowledge. Results show that models consistently outperform chance but remain far from saturation, exhibiting pronounced sensitivity to temporal composition and reasoning order, as well as systematic failures on precise temporal localization and multi-condition symbolic judgments.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 9 references that are in the map. First posted on arXiv 13 Feb 2026.
Builds on
9 of 9 references.
Closest ancestors · 1
Same direction, newest first
- 2025BaZi-Based Character Simulation Benchmark: Evaluating AI on Temporal and Persona Reasoning27 Oct
Other references · 8
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
- 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 13
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models
- 2026LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models
- 2026QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
- 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
- 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
- 2026LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
- 2025NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Models
- 2025The Art of Scaling Test-Time Compute for Large Language Models