# TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training

arXiv: https://arxiv.org/abs/2104.01099 · HTML: https://mlsanity.com/paper/2104.01099/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [INFOTABS: Inference on Tables as Semi-structured Data](https://mlsanity.com/paper/2005.06117/) (2020)
- [Understanding tables with intermediate pre-training](https://mlsanity.com/paper/2010.00571/) (2020)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [The Fast and the Flexible: training neural networks to learn to follow instructions from small data](https://mlsanity.com/paper/1809.06194/) (2018)

## Built on by (most cited first, 14 of 14)

- [PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training](https://mlsanity.com/paper/2211.02816/) (2022)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)](https://mlsanity.com/paper/2105.13995/) (2021)
- [TempTabQA: Temporal Question Answering for Semi-Structured Tables](https://mlsanity.com/paper/2311.08002/) (2023)
- [Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning](https://mlsanity.com/paper/2108.00578/) (2021)
- [Knowledge-Aware Reasoning over Multimodal Semi-structured Tables](https://mlsanity.com/paper/2408.13860/) (2024)
- [Realistic Data Augmentation Framework for Enhancing Tabular Reasoning](https://mlsanity.com/paper/2210.12795/) (2022)
- [Enhancing Tabular Reasoning with Pattern Exploiting Training](https://mlsanity.com/paper/2210.12259/) (2022)
- [Leveraging Data Recasting to Enhance Tabular Reasoning](https://mlsanity.com/paper/2211.12641/) (2022)
- [InfoSync: Information Synchronization across Multilingual Semi-structured Tables](https://mlsanity.com/paper/2307.03313/) (2023)
- [DataTales: A Benchmark for Real-World Intelligent Data Narration](https://mlsanity.com/paper/2410.17859/) (2024)
- [Plan-then-Seam: Towards Efficient Table-to-Text Generation](https://mlsanity.com/paper/2302.05138/) (2023)
- [Agentic DAG-Orchestrated Planner Framework for Multi-Modal, Multi-Hop Question Answering in Hybrid Data Lakes](https://mlsanity.com/paper/2603.14229/) (2026)
- [A Multi-Agent Approach for Claim Verification from Tabular Data Documents](https://mlsanity.com/paper/2604.17225/) (2026)

## Abstract (arXiv)

We present the TAPAS contribution to the Shared Task on Statement Verification and Evidence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)). SEM TAB FACT Task A is a classification task of recognizing if a statement is entailed, neutral or refuted by the content of a given table. We adopt the binary TAPAS model of Eisenschlos et al. (2020) to this task. We learn two binary classification models: A first model to predict if a statement is neutral or non-neutral and a second one to predict if it is entailed or refuted. As the shared task training set contains only entailed or refuted examples, we generate artificial neutral examples to train the first model. Both models are pre-trained using a MASKLM objective, intermediate counter-factual and synthetic data (Eisenschlos et al., 2020) and TABFACT (Chen et al., 2020), a large table entailment dataset. We find that the artificial neutral examples are somewhat effective at training the first model, achieving 68.03 test F1 versus the 60.47 of a majority baseline. For the second stage, we find that the pre-training on the intermediate data and TABFACT improves the results over MASKLM pre-training (68.03 vs 57.01).