Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models
On the map: LLM reasoning → Knowledge- and tool-augmented reasoning
Abstract (arXiv)
This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) utilizing step-by-step reasoning methods in the TableQA task. The method processes tasks by sequentially identifying critical columns and rows given question and table with its structure, determining necessary aggregations, calculations, or comparisons, and finally inferring the results to generate a precise prediction. By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA, while smaller LMs like Llama-2-7B shows an 11% performance decline. We empirically investigate the potential causes of performance contrasts by exploring the capabilities of bigger and smaller LMs from various dimensions in TableQA task. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 18 references that are in the map. First posted on arXiv 24 Nov 2024.
Builds on
16 of 18 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Qwen Technical Reportcited by 3,344
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
- 2023PaLM 2 Technical Reportcited by 1,162
- 2022Large Language Models Can Self-Improvecited by 664
- 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
- 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
3 more
- 2024How to think step-by-step: A mechanistic understanding of chain-of-thought reasoningcited by 53
- 2023Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarioscited by 12
- 2023TableQAKit: A Comprehensive and Practical Toolkit for Table-based Question Answeringcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Large Language Models are few(1)-shot Table Reasoners
- 2024A NotSo Simple Way to Beat Simple Bench
- 2024A Survey of Table Reasoning with Large Language Models
- 2025Towards Reasoning Ability of Small Language Models
- 2023Effective Distillation of Table-based Reasoning Ability from LLMs
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
- 2025TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models
- 2024Table Question Answering for Low-resourced Indic Languages