mlsanityData to 7 Oct 2026

Annotating Columns with Pre-trained Language Models

arXiv 2104.01785 ↗First posted 5 Apr 2021cs.DB cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Inferring meta information about tables, such as column headers or relationships between columns, is an active research topic in data management as we find many tables are missing some of this information. In this paper, we study the problem of annotating table columns (i.e., predicting column types and the relationships between columns) using only information from the table itself. We develop a multi-task learning framework (called Doduo) based on pre-trained language models, which takes the entire table as input and predicts column types/relations using a single model. Experimental results show that Doduo establishes new state-of-the-art performance on two benchmarks for the column type prediction and column relation prediction tasks with up to 4.0% and 11.9% improvements, respectively. We report that Doduo can already outperform the previous state-of-the-art performance with a minimal number of tokens, only 8 tokens per column. We release a toolbox (https://github.com/megagonlabs/doduo) and confirm the effectiveness of Doduo on a real-world data science problem through a case study.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 34 of its 71 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

34 of 71 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
22 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  2. 2020Efficient Transformers: A Surveycited by 784
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  5. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  6. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  7. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  8. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  9. 2020Deep Entity Matching with Pre-Trained Language Modelscited by 143
  10. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  11. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  12. 2020TUTA: Tree-based Transformers for Generally Structured Table Pre-trainingcited by 100
  13. 2019Sherlock: A Deep Learning Approach to Semantic Data Type Detectioncited by 73
  14. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  15. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  16. 2019Dataset search: a surveycited by 36
  17. 2020Valentine: Evaluating Matching Techniques for Dataset Discoverycited by 34
  18. 2019VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repositorycited by 28
  19. 2020RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparationcited by 27
  20. 2021TCN: Table Convolutional Network for Web Table Interpretationcited by 24
  21. 2019Learning Semantic Annotations for Tabular Datacited by 17
  22. 2020Semantic Labeling Using a Deep Contextualized Language Modelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 8; 2025: 10; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  2. 2023Table-GPT: Table-tuned GPT for Diverse Table Taskscited by 69
  3. 2023Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMscited by 49
  4. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 35
  5. 2023CHORUS: Foundation Models for Unified Data Discovery and Explorationcited by 24
  6. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  7. 2023Column Type Annotation using ChatGPTcited by 18
  8. 2023WDC Products: A Multi-Dimensional Entity Matching Benchmarkcited by 17
  9. 2022DeepJoin: Joinable Table Discovery with Pre-trained Language Modelscited by 15
  10. 2024Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AIcited by 15
  11. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  12. 2025MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmarkcited by 13
25 more
  1. 2023Observatory: Characterizing Embeddings of Relational Tablescited by 12
  2. 2023Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Taskscited by 10
  3. 2023LakeBench: Benchmarks for Data Discovery over Data Lakescited by 9
  4. 2024RACOON: An LLM-based Framework for Retrieval-Augmented Column Type Annotation with a Knowledge Graphcited by 7
  5. 2024Mind the Data Gap: Bridging LLMs to Enterprise Data Integrationcited by 7
  6. 2025AutoG: Towards automatic graph construction from tabular datacited by 6
  7. 2025Columbo: Expanding Abbreviated Column Names for Tabular Data Using Large Language Modelscited by 5
  8. 2023AdaTyper: Adaptive Semantic Column Type Detectioncited by 4
  9. 2025Snoopy: Effective and Efficient Semantic Join Discovery via Proxy Columnscited by 4
  10. 2023Adversarial Attacks on Tables with Entity Swapcited by 3
  11. 2024KGLink: A column type annotation method that combines knowledge graph and pre-trained language modelcited by 3
  12. 2023Embeddings for Tabular Data: A Surveycited by 2
  13. 2024Evaluating LLMs on Entity Disambiguation in Tablescited by 2
  14. 2025Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tablescited by 2
  15. 2025Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligencecited by 2
  16. 2024TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakescited by 1
  17. 2024Leveraging Foundation Language Models (FLMs) for Automated Cohort Extraction from Large EHR Databasescited by 1
  18. 2026TACO: Task-Aware Column Description Generation Using LLMscited by 1
  19. 2023Semantic Data Management in Data Lakescited by 0
  20. 2024T-KAER: Transparency-enhanced Knowledge-Augmented Entity Resolution Frameworkcited by 0
  21. 2025PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Modelscited by 0
  22. 2025Language Models That Walk the Talk: A Framework for Formal Fairness Certificatescited by 0
  23. 2025ZeroCard: Cardinality Estimation with Zero Dependence on Target Databases -- No Data, No Query, No Retrainingcited by 0
  24. 2026ZTab: Domain-based Zero-shot Annotation for Table Columnscited by 0
  25. 2026An Explainable Header-Centric Framework for Large-Scale Semantic Table Interpretation and Data Quality Assessmentcited by 0