# SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation

arXiv: https://arxiv.org/abs/2603.08910 · HTML: https://mlsanity.com/paper/2603.08910/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation](https://mlsanity.com/paper/2108.06712/) (2021)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables](https://mlsanity.com/paper/2305.13186/) (2023)
- [AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry](https://mlsanity.com/paper/2106.12944/) (2021)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://mlsanity.com/paper/2311.02216/) (2023)
- [Towards Cognitive AI Systems: a Survey and Prospective on Neuro-Symbolic AI](https://mlsanity.com/paper/2401.01040/) (2024)
- [Table-R1: Inference-Time Scaling for Table Reasoning](https://mlsanity.com/paper/2505.23621/) (2025)
- [TQA-Bench: Evaluating LLMs for Multi-Table Question Answering](https://mlsanity.com/paper/2411.19504/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering](https://mlsanity.com/paper/2506.03949/) (2025)
- [SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types](https://mlsanity.com/paper/2412.11757/) (2024)
- [Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges](https://mlsanity.com/paper/2508.00217/) (2025)
- [DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation](https://mlsanity.com/paper/2403.02528/) (2024)
- [LLM-Symbolic Integration for Robust Temporal Tabular Reasoning](https://mlsanity.com/paper/2506.05746/) (2025)
- [MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table Reasoning](https://mlsanity.com/paper/2506.05813/) (2025)
- [TableReasoner: Advancing Table Reasoning Framework with Large Language Models](https://mlsanity.com/paper/2507.08046/) (2025)
- [PanelTR: Zero-Shot Table Reasoning Framework Through Multi-Agent Scientific Discussion](https://mlsanity.com/paper/2508.06110/) (2025)

## Abstract (arXiv)

We introduce SciTaRC, an expert-authored benchmark for question answering over scientific tables that targets composite, multi-step reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table reasoning.