# AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data

arXiv: https://arxiv.org/abs/2507.18442 · HTML: https://mlsanity.com/paper/2507.18442/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 45 of its 57 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 45)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability](https://mlsanity.com/paper/2402.18667/) (2024)
- [TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios](https://mlsanity.com/paper/2403.19318/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion](https://mlsanity.com/paper/2409.14051/) (2024)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [Tab-CoT: Zero-shot Tabular Chain of Thought](https://mlsanity.com/paper/2305.17812/) (2023)
- [Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table](https://mlsanity.com/paper/2305.07288/) (2023)
- [Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs](https://mlsanity.com/paper/2402.12424/) (2024)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [Rethinking Tabular Data Understanding with Large Language Models](https://mlsanity.com/paper/2312.16702/) (2023)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation](https://mlsanity.com/paper/2406.12975/) (2024)
- [PRE: A Peer Review Based Large Language Model Evaluator](https://mlsanity.com/paper/2401.15641/) (2024)

## Built on by (most cited first, 2 of 2)

- [Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps](https://mlsanity.com/paper/2510.13430/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)

## Abstract (arXiv)

The cognitive and reasoning abilities of large language models (LLMs) have enabled remarkable progress in natural language processing. However, their performance in interpreting structured data, especially in tabular formats, remains limited. Although benchmarks for English tabular data are widely available, Arabic is still underrepresented because of the limited availability of public resources and its unique language features. To address this gap, we present AraTable, a novel and comprehensive benchmark designed to evaluate the reasoning and understanding capabilities of LLMs when applied to Arabic tabular data. AraTable consists of various evaluation tasks, such as direct question answering, fact verification, and complex reasoning, involving a wide range of Arabic tabular sources. Our methodology follows a hybrid pipeline, where initial content is generated by LLMs and subsequently filtered and verified by human experts to ensure high dataset quality. Initial analyses using AraTable show that, while LLMs perform adequately on simpler tabular tasks such as direct question answering, they continue to face significant cognitive challenges when tasks require deeper reasoning and fact verification. This indicates that there are substantial opportunities for future work to improve performance on complex tabular reasoning tasks. We also propose a fully automated evaluation framework that uses a self-deliberation mechanism and achieves performance nearly identical to that of human judges. This research provides a valuable, publicly available resource and evaluation framework that can help accelerate the development of foundational models for processing and analysing Arabic structured data.