mlsanityData to 7 Oct 2026

TAPEX: Table Pre-training via Learning a Neural SQL Executor

arXiv 2107.07653 ↗First posted 16 Jul 2021cs.CL cs.AIcited by 233

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

39 of 51 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  8. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  9. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  10. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  11. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  12. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
27 more
  1. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  2. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  3. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  4. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  5. 2020TUTA: Tree-based Transformers for Generally Structured Table Pre-trainingcited by 100
  6. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  7. 2020Grounded Adaptation for Zero-shot Executable Semantic Parsingcited by 76
  8. 2020Understanding tables with intermediate pre-trainingcited by 68
  9. 2020Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Trainingcited by 65
  10. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  11. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 59
  12. 2018Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsingcited by 57
  13. 2020How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Contextcited by 53
  14. 2018Robust Text-to-SQL Generation with Execution-Guided Decodingcited by 51
  15. 2021Learning to Synthesize Data for Semantic Parsingcited by 41
  16. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  17. 2015Neural Programmer: Inducing Latent Programs with Gradient Descentcited by 32
  18. 2020Program Enhanced Fact Verification with Verbalization and Graph Attention Networkcited by 29
  19. 2019Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programscited by 26
  20. 2020LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Networkcited by 26
  21. 2019Answering Conversational Questions on Structured Data without Logical Formscited by 24
  22. 2016Learning a Natural Language Interface with Neural Programmercited by 14
  23. 2017Weakly-supervised Semantic Parsing with Abstract Examplescited by 13
  24. 2017Macro Grammars and Holistic Triggering for Efficient Semantic Parsingcited by 10
  25. 2018Knowledge-Aware Conversational Semantic Parsing Over Web Tablescited by 6
  26. 2019A Split-and-Recombine Approach for Follow-up Query Analysiscited by 2
  27. 2019Using Database Rule for Weak Supervised Text-to-SQL Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 38; 2023: 49; 2024: 56; 2025: 55; 2026: 34.

233 papers.

Sort:

By area:

  1. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  2. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  3. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  4. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  6. 2022Binding Language Models in Symbolic Languagescited by 224
  7. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  8. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  9. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  10. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  11. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  12. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
38 more
  1. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  2. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  3. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  4. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  5. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 90
  6. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  7. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 76
  8. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  9. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  10. 2024Large Language Model for Table Processing: A Surveycited by 52
  11. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  12. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  13. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tablescited by 51
  14. 2022Reasoning Like Program Executorscited by 45
  15. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  16. 2024Multimodal Table Understandingcited by 43
  17. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  18. 2022PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-trainingcited by 41
  19. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 41
  20. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 41
  21. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  22. 2023Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queriescited by 38
  23. 2023Rethinking Tabular Data Understanding with Large Language Modelscited by 34
  24. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  25. 2023MultiTabQA: Generating Tabular Answers for Multi-Table Question Answeringcited by 32
  26. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoningcited by 32
  27. 2025Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table Reasoningcited by 32
  28. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  29. 2022OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answeringcited by 29
  30. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examplescited by 29
  31. 2024StructLM: Towards Building Generalist Models for Structured Knowledge Groundingcited by 29
  32. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answeringcited by 25
  33. 2024OpenTab: Advancing Large Language Models as Open-domain Table Reasonerscited by 25
  34. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  35. 2023Effective Distillation of Table-based Reasoning Ability from LLMscited by 23
  36. 2024CABINET: Content Relevance based Noise Reduction for Table Question Answeringcited by 23
  37. 2024ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Contextcited by 23
  38. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22

50 most cited of 233 All 233 (JSON)