# PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training

arXiv: https://arxiv.org/abs/2211.02816 · HTML: https://mlsanity.com/paper/2211.02816/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 5 Nov 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Understanding tables with intermediate pre-training](https://mlsanity.com/paper/2010.00571/) (2020)
- [On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries](https://mlsanity.com/paper/2010.11246/) (2020)
- [SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)](https://mlsanity.com/paper/2105.13995/) (2021)
- [Program Enhanced Fact Verification with Verbalization and Graph Attention Network](https://mlsanity.com/paper/2010.03084/) (2020)
- [LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network](https://mlsanity.com/paper/2004.13659/) (2020)
- [Table Search Using a Deep Contextualized Language Model](https://mlsanity.com/paper/2005.09207/) (2020)
- [Joint Verification and Reranking for Open Fact Checking Over Tables](https://mlsanity.com/paper/2012.15115/) (2020)
- [Detecting Incongruity Between News Headline and Body Text via a Deep Hierarchical Encoder](https://mlsanity.com/paper/1811.07066/) (2018)
- [TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training](https://mlsanity.com/paper/2104.01099/) (2021)
- [Table-based Fact Verification with Self-adaptive Mixture of Experts](https://mlsanity.com/paper/2204.08753/) (2022)
- [Table-based Fact Verification with Salience-aware Learning](https://mlsanity.com/paper/2109.04053/) (2021)
- [Volta at SemEval-2021 Task 9: Statement Verification and Evidence Finding with Tables using TAPAS and Transfer Learning](https://mlsanity.com/paper/2106.00248/) (2021)

## Built on by (most cited first, 30 of 41)

- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables](https://mlsanity.com/paper/2305.13186/) (2023)
- [TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition](https://mlsanity.com/paper/2404.10150/) (2024)
- [Rethinking Tabular Data Understanding with Large Language Models](https://mlsanity.com/paper/2312.16702/) (2023)
- [MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering](https://mlsanity.com/paper/2305.12820/) (2023)
- [CABINET: Content Relevance based Noise Reduction for Table Question Answering](https://mlsanity.com/paper/2402.01155/) (2024)
- [ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context](https://mlsanity.com/paper/2403.02177/) (2024)
- [TableMaster: A Recipe to Advance Table Understanding with Language Models](https://mlsanity.com/paper/2501.19378/) (2025)
- [TQA-Bench: Evaluating LLMs for Multi-Table Question Answering](https://mlsanity.com/paper/2411.19504/) (2024)
- [VerifAI: Verified Generative AI](https://mlsanity.com/paper/2307.02796/) (2023)
- [Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement](https://mlsanity.com/paper/2402.00745/) (2024)
- [NormTab: Improving Symbolic Reasoning in LLMs Through Tabular Data Normalization](https://mlsanity.com/paper/2406.17961/) (2024)
- [Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments](https://mlsanity.com/paper/2212.09683/) (2022)
- [From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuning](https://mlsanity.com/paper/2304.07995/) (2023)
- [Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretraining](https://mlsanity.com/paper/2403.20208/) (2024)
- [AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework](https://mlsanity.com/paper/2412.10422/) (2024)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning](https://mlsanity.com/paper/2409.11724/) (2024)
- [When TableQA Meets Noise: A Dual Denoising Framework for Complex Questions and Large-scale Tables](https://mlsanity.com/paper/2509.17680/) (2025)
- [Rethinking Table Pruning in TableQA: From Sequential Revisions to Gold Trajectory-Supervised Parallel Search](https://mlsanity.com/paper/2601.03851/) (2026)
- [MFORT-QA: Multi-hop Few-shot Open Rich Table Question Answering](https://mlsanity.com/paper/2403.19116/) (2024)
- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [RePanda: Pandas-powered Tabular Verification and Reasoning](https://mlsanity.com/paper/2503.11921/) (2025)
- [HeLM: Highlighted Evidence augmented Language Model for Enhanced Table-to-Text Generation](https://mlsanity.com/paper/2311.08896/) (2023)
- [MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table Reasoning](https://mlsanity.com/paper/2506.05813/) (2025)
- [iTBLS: A Dataset of Interactive Conversations Over Tabular Information](https://mlsanity.com/paper/2404.12580/) (2024)
- [DataTales: A Benchmark for Real-World Intelligent Data Narration](https://mlsanity.com/paper/2410.17859/) (2024)
- [TabVer: Tabular Fact Verification with Natural Logic](https://mlsanity.com/paper/2411.01093/) (2024)

## Abstract (arXiv)

Fact verification has attracted a lot of research attention recently, e.g., in journalism, marketing, and policymaking, as misinformation and disinformation online can sway one's opinion and affect one's actions. While fact-checking is a hard task in general, in many cases, false statements can be easily debunked based on analytics over tables with reliable information. Hence, table-based fact verification has recently emerged as an important and growing research area. Yet, progress has been limited due to the lack of datasets that can be used to pre-train language models (LMs) to be aware of common table operations, such as aggregating a column or comparing tuples. To bridge this gap, in this paper we introduce PASTA, a novel state-of-the-art framework for table-based fact verification via pre-training with synthesized sentence-table cloze questions. In particular, we design six types of common sentence-table cloze tasks, including Filter, Aggregation, Superlative, Comparative, Ordinal, and Unique, based on which we synthesize a large corpus consisting of 1.2 million sentence-table pairs from WikiTables. PASTA uses a recent pre-trained LM, DeBERTaV3, and further pretrains it on our corpus. Our experimental results show that PASTA achieves new state-of-the-art performance on two table-based fact verification benchmarks: TabFact and SEM-TAB-FACTS. In particular, on the complex set of TabFact, which contains multiple operations, PASTA largely outperforms the previous state of the art by 4.7 points (85.6% vs. 80.9%), and the gap between PASTA and human performance on the small TabFact test set is narrowed to just 1.5 points (90.6% vs. 92.1%).