# Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning

arXiv: https://arxiv.org/abs/2305.07475 · HTML: https://mlsanity.com/paper/2305.07475/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation](https://mlsanity.com/paper/2108.06712/) (2021)
- [NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://mlsanity.com/paper/2204.05660/) (2022)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Joint entity recognition and relation extraction as a multi-head selection problem](https://mlsanity.com/paper/1804.07847/) (2018)
- [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Span Selection Pre-training for Question Answering](https://mlsanity.com/paper/1909.04120/) (2019)
- [MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving](https://mlsanity.com/paper/2107.13435/) (2021)
- [OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering](https://mlsanity.com/paper/2207.03637/) (2022)
- [The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/2002.07972/) (2020)
- [Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation](https://mlsanity.com/paper/2212.09994/) (2022)
- [ReasonBERT: Pre-trained to Reason with Distant Supervision](https://mlsanity.com/paper/2109.04912/) (2021)
- [Adversarial Examples for Evaluating Math Word Problem Solvers](https://mlsanity.com/paper/2109.05925/) (2021)
- [FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining](https://mlsanity.com/paper/2109.07323/) (2021)
- [ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler](https://mlsanity.com/paper/2210.10105/) (2022)

## Abstract (arXiv)

Numerical reasoning over table-and-text hybrid passages, such as financial reports, poses significant challenges and has numerous potential applications. Noise and irrelevant variables in the model input have been a hindrance to its performance. Additionally, coarse-grained supervision of the whole solution program has impeded the model's ability to learn the underlying numerical reasoning process. In this paper, we propose three pretraining tasks that operate at both the whole program and sub-program level: Variable Integrity Ranking, which guides the model to focus on useful variables; Variable Operator Prediction, which decomposes the supervision into fine-grained single operator prediction; and Variable Keyphrase Masking, which encourages the model to identify key evidence that sub-programs are derived from. Experimental results demonstrate the effectiveness of our proposed methods, surpassing transformer-based model baselines.