mlsanityData to 7 Oct 2026

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

arXiv 2005.08314 ↗First posted 17 May 2020cs.CL cs.LGcited by 395

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 61; 2023: 74; 2024: 78; 2025: 68; 2026: 56.

395 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  4. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  5. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  6. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  7. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  9. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  10. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  11. 2022Binding Language Models in Symbolic Languagescited by 224
  12. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
38 more
  1. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  2. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  3. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  4. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  5. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  6. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  7. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  8. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 132
  9. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  10. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  11. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
  12. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  13. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  14. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  15. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 101
  16. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  17. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 90
  18. 2024KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graphcited by 90
  19. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 90
  20. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  21. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  22. 2023CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesiscited by 78
  23. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 76
  24. 2023Table-GPT: Table-tuned GPT for Diverse Table Taskscited by 69
  25. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  26. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  27. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  28. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  29. 2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
  30. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  31. 2022Transfer Learning with Deep Tabular Modelscited by 52
  32. 2024Large Language Model for Table Processing: A Surveycited by 52
  33. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  34. 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
  35. 2021GitTables: A Large-Scale Corpus of Relational Tablescited by 44
  36. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44
  37. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  38. 2021SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQLcited by 43

50 most cited of 395 All 395 (JSON)