# FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

arXiv: https://arxiv.org/abs/2109.07323 · HTML: https://mlsanity.com/paper/2109.07323/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation](https://mlsanity.com/paper/2108.06712/) (2021)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [Understanding tables with intermediate pre-training](https://mlsanity.com/paper/2010.00571/) (2020)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing](https://mlsanity.com/paper/1807.02322/) (2018)
- [Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension](https://mlsanity.com/paper/1909.00109/) (2019)
- [SpreadsheetCoder: Formula Prediction from Semi-structured Context](https://mlsanity.com/paper/2106.15339/) (2021)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation](https://mlsanity.com/paper/2012.02469/) (2020)
- [Retrieving Complex Tables with Multi-Granular Graph Representation Learning](https://mlsanity.com/paper/2105.01736/) (2021)
- [NumGPT: Improving Numeracy Ability of Generative Pre-trained Models](https://mlsanity.com/paper/2109.03137/) (2021)
- [Learning to Reason for Text Generation from Scientific Tables](https://mlsanity.com/paper/2104.08296/) (2021)
- [TableSense: Spreadsheet Table Detection with Convolutional Neural Networks](https://mlsanity.com/paper/2106.13500/) (2021)

## Built on by (most cited first, 19 of 19)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering](https://mlsanity.com/paper/2305.12820/) (2023)
- [TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data](https://mlsanity.com/paper/2205.12682/) (2022)
- [SpreadsheetLLM: Encoding Spreadsheets for Large Language Models](https://mlsanity.com/paper/2407.09025/) (2024)
- [NameGuess: Column Name Expansion for Tabular Data](https://mlsanity.com/paper/2310.13196/) (2023)
- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections](https://mlsanity.com/paper/2508.15851/) (2025)
- [FormaT5: Abstention and Examples for Conditional Table Formatting with Natural Language](https://mlsanity.com/paper/2310.17306/) (2023)
- [Table Question Answering for Low-resourced Indic Languages](https://mlsanity.com/paper/2410.03576/) (2024)
- [General Table Question Answering via Answer-Formula Joint Generation](https://mlsanity.com/paper/2503.12345/) (2025)
- [Bridge the Gap between Language models and Tabular Understanding](https://mlsanity.com/paper/2302.09302/) (2023)
- [NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answering](https://mlsanity.com/paper/2211.03462/) (2022)
- [QTSumm: Query-Focused Summarization over Tabular Data](https://mlsanity.com/paper/2305.14303/) (2023)
- [Formula-R1: Incentivizing LLM Reasoning over Complex Tables with Numerical Computation via Formula-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.23667/) (2025)
- [Training Table Question Answering via SQL Query Decomposition](https://mlsanity.com/paper/2402.13288/) (2024)
- [FACTS: Table Summarization via Offline Template Generation with Agentic Workflows](https://mlsanity.com/paper/2510.13920/) (2025)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation](https://mlsanity.com/paper/2607.19354/) (2026)
- [Structured Prediction for Scalable Spreadsheet Table Understanding: From Cell Types to Table Ranges (Extended Version)](https://mlsanity.com/paper/2608.16050/) (2026)

## Abstract (arXiv)

Tables store rich numerical data, but numerical reasoning over tables is still a challenge. In this paper, we find that the spreadsheet formula, which performs calculations on numerical values in tables, is naturally a strong supervision of numerical reasoning. More importantly, large amounts of spreadsheets with expert-made formulae are available on the web and can be obtained easily. FORTAP is the first method for numerical-reasoning-aware table pretraining by leveraging large corpus of spreadsheet formulae. We design two formula pretraining tasks to explicitly guide FORTAP to learn numerical reference and calculation in semi-structured tables. FORTAP achieves state-of-the-art results on two representative downstream tasks, cell type classification and formula prediction, showing great potential of numerical-reasoning-aware pretraining.