mlsanityData to 7 Oct 2026

TABBIE: Pretrained Representations of Tabular Data

arXiv 2105.02584 ↗First posted 6 May 2021cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

Existing work on tabular representation learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model's learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

20 of 36 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  10. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  11. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  12. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
8 more
  1. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  2. 2019Sherlock: A Deep Learning Approach to Semantic Data Type Detectioncited by 73
  3. 2018Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsingcited by 57
  4. 2020Table Structure Recognition using Top-Down and Bottom-Up Cuescited by 53
  5. 2019Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrievalcited by 37
  6. 2019VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repositorycited by 28
  7. 2017EntiTables: Smart Assistance for Entity-Focused Tablescited by 11
  8. 2020Novel Entity Discovery from Web Tablescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 23; 2023: 30; 2024: 22; 2025: 20; 2026: 10.

111 papers.

Sort:

By area:

  1. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  4. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  5. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  6. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  7. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  8. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 76
  11. 2024Large Language Model for Table Processing: A Surveycited by 52
  12. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
38 more
  1. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 35
  2. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  3. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoningcited by 32
  4. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30
  5. 2023TempTabQA: Temporal Question Answering for Semi-Structured Tablescited by 27
  6. 2023Schema-Driven Information Extraction from Heterogeneous Tablescited by 24
  7. 2023CHORUS: Foundation Models for Unified Data Discovery and Explorationcited by 24
  8. 2022SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representationcited by 23
  9. 2023TabLib: A Dataset of 627M Tables with Contextcited by 23
  10. 2024ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Contextcited by 23
  11. 2021ReasonBERT: Pre-trained to Reason with Distant Supervisioncited by 22
  12. 2025Large Language Models are Good Relational Learnerscited by 22
  13. 2022Table-To-Text generation and pre-training with TabT5cited by 20
  14. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  15. 2021FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretrainingcited by 19
  16. 2023MotherNet: Fast Training and Inference via Hyper-Network Transformerscited by 19
  17. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  18. 2022Tele-Knowledge Pre-training for Fault Analysiscited by 18
  19. 2023Column Type Annotation using ChatGPTcited by 18
  20. 2022DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articlescited by 17
  21. 2025RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarkingcited by 17
  22. 2022PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generationcited by 16
  23. 2022DeepJoin: Joinable Table Discovery with Pre-trained Language Modelscited by 15
  24. 2023NameGuess: Column Name Expansion for Tabular Datacited by 13
  25. 2023SUQL: Conversational Search over Structured and Unstructured Data with Large Language Modelscited by 13
  26. 2022Table Retrieval May Not Necessitate Table-specific Model Designcited by 11
  27. 2023Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Taskscited by 10
  28. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  29. 2021Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoningcited by 9
  30. 2023LakeBench: Benchmarks for Data Discovery over Data Lakescited by 9
  31. 2024Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarkingcited by 9
  32. 2021CpT: Convolutional Point Transformer for 3D Point Cloud Processingcited by 8
  33. 2022AnaMeta: A Table Understanding Dataset of Field Metadata Knowledge Shared by Multi-dimensional Data Analysis Taskscited by 8
  34. 2023Reconstructing Materials Tetrahedron: Challenges in Materials Information Extractioncited by 8
  35. 2024Enhancing Temporal Understanding in LLMs for Semi-structured Tablescited by 8
  36. 2023NCART: Neural Classification and Regression Tree for Tabular Datacited by 7
  37. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  38. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6

50 most cited of 111 All 111 (JSON)