# HYTREL: Hypergraph-enhanced Tabular Data Representation Learning

arXiv: https://arxiv.org/abs/2307.08623 · HTML: https://mlsanity.com/paper/2307.08623/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 53 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks](https://mlsanity.com/paper/2106.13264/) (2021)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://mlsanity.com/paper/2203.00274/) (2022)
- [On the Benefits of Invariance in Neural Networks](https://mlsanity.com/paper/2005.00178/) (2020)
- [Sherlock: A Deep Learning Approach to Semantic Data Type Detection](https://mlsanity.com/paper/1905.10688/) (2019)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [HyperSAGE: Generalizing Inductive Representation Learning on Hypergraphs](https://mlsanity.com/paper/2010.04558/) (2020)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [Annotating Columns with Pre-trained Language Models](https://mlsanity.com/paper/2104.01785/) (2021)
- [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://mlsanity.com/paper/2205.03972/) (2022)
- [Capturing Row and Column Semantics in Transformer Based Question Answering over Tables](https://mlsanity.com/paper/2104.08303/) (2021)

## Built on by (most cited first, 30 of 36)

- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Uncertainty Quantification for In-Context Learning of Large Language Models](https://mlsanity.com/paper/2402.10189/) (2024)
- [Multimodal Table Understanding](https://mlsanity.com/paper/2406.08100/) (2024)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning](https://mlsanity.com/paper/2410.07074/) (2024)
- [RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking](https://mlsanity.com/paper/2504.01346/) (2025)
- [Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AI](https://mlsanity.com/paper/2407.21523/) (2024)
- [Graph Neural Networks for Tabular Data Learning: A Survey with Taxonomy and Directions](https://mlsanity.com/paper/2401.02143/) (2024)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing](https://mlsanity.com/paper/2501.02191/) (2025)
- [HyperG: Hypergraph-Enhanced LLMs for Structured Knowledge](https://mlsanity.com/paper/2502.18125/) (2025)
- [UniPredict: Large Language Models are Universal Tabular Classifiers](https://mlsanity.com/paper/2310.03266/) (2023)
- [ItD: Large Language Models Can Teach Themselves Induction through Deduction](https://mlsanity.com/paper/2403.05789/) (2024)
- [Table as a Modality for Large Language Models](https://mlsanity.com/paper/2512.00947/) (2025)
- [Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction](https://mlsanity.com/paper/2411.04219/) (2024)
- [Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks](https://mlsanity.com/paper/2604.21696/) (2026)
- [TabSeq: A Framework for Deep Learning on Tabular Data via Sequential Ordering](https://mlsanity.com/paper/2410.13203/) (2024)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes](https://mlsanity.com/paper/2502.02672/) (2025)
- [2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models](https://mlsanity.com/paper/2409.19700/) (2024)
- [Enhancing Table Representations with LLM-powered Synthetic Data Generation](https://mlsanity.com/paper/2411.03356/) (2024)
- [Piece of Table: A Divide-and-Conquer Approach for Selecting Subtables in Table Question Answering](https://mlsanity.com/paper/2412.07629/) (2024)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization](https://mlsanity.com/paper/2502.18847/) (2025)
- [RAISE: Reinforced Adaptive Instruction Selection For Large Language Models](https://mlsanity.com/paper/2504.07282/) (2025)
- [Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nesting](https://mlsanity.com/paper/2502.15819/) (2025)
- [Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks](https://mlsanity.com/paper/2505.21329/) (2025)
- [The Power of Order: Fooling LLMs with Adversarial Table Permutations](https://mlsanity.com/paper/2605.00445/) (2026)
- [DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data](https://mlsanity.com/paper/2605.03430/) (2026)
- [TEmBed-T: A Multi-Dimensional Benchmark for Table-Level Embeddings](https://mlsanity.com/paper/2607.24130/) (2026)

## Abstract (arXiv)

Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do not take into account the row/column permutation invariances, hierarchical structure, etc. that exist in tabular data. To alleviate these limitations, we propose HYTREL, a tabular language model, that captures the permutation invariances and three more structural properties of tabular data by using hypergraphs - where the table cells make up the nodes and the cells occurring jointly together in each row, column, and the entire table are used to form three different types of hyperedges. We show that HYTREL is maximally invariant under certain conditions for tabular data, i.e., two tables obtain the same representations via HYTREL iff the two tables are identical up to permutations. Our empirical results demonstrate that HYTREL consistently outperforms other competitive baselines on four downstream tasks with minimal pretraining, illustrating the advantages of incorporating the inductive biases associated with tabular data into the representations. Finally, our qualitative analyses showcase that HYTREL can assimilate the table structures to generate robust representations for the cells, rows, columns, and the entire table.