# Innovative tokenisation of structured data for LLM training

arXiv: https://arxiv.org/abs/2508.01685 · HTML: https://mlsanity.com/paper/2508.01685/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios](https://mlsanity.com/paper/2403.19318/) (2024)
- [Large Language Model for Table Processing: A Survey](https://mlsanity.com/paper/2402.05121/) (2024)
- [Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs](https://mlsanity.com/paper/2402.12424/) (2024)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization](https://mlsanity.com/paper/2405.17067/) (2024)
- [A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models](https://mlsanity.com/paper/2010.15036/) (2020)
- [An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning](https://mlsanity.com/paper/1611.08331/) (2016)
- [A Comprehensive Survey of Synthetic Tabular Data Generation](https://mlsanity.com/paper/2504.16506/) (2025)
- [Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System](https://mlsanity.com/paper/2504.09207/) (2025)
- [Curriculum Graph Machine Learning: A Survey](https://mlsanity.com/paper/2302.02926/) (2023)
- [Evaluating Various Tokenizers for Arabic Text Classification](https://mlsanity.com/paper/2106.07540/) (2021)
- [LLM Embeddings for Deep Learning on Tabular Data](https://mlsanity.com/paper/2502.11596/) (2025)
- [Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions](https://mlsanity.com/paper/2501.03540/) (2025)
- [Tabular Embedding Model (TEM): Finetuning Embedding Models For Tabular RAG Applications](https://mlsanity.com/paper/2405.01585/) (2024)
- [Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers](https://mlsanity.com/paper/2411.01645/) (2024)
- [A Survey of Text Representation Methods and Their Genealogy](https://mlsanity.com/paper/2211.14591/) (2022)
- [Scalable Representation Learning for Multimodal Tabular Transactions](https://mlsanity.com/paper/2410.07851/) (2024)
- [Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)](https://mlsanity.com/paper/2410.03568/) (2024)

## Abstract (arXiv)

Data representation remains a fundamental challenge in machine learning, particularly when adapting sequence-based architectures like Transformers and Large Language Models (LLMs) for structured tabular data. Existing methods often fail to cohesively encode the mix of numerical and categorical features or preserve the inherent structure of tables. This paper introduces a novel, hybrid tokenisation methodology designed to convert tabular data into a unified, sequential format suitable for LLM training. Our approach combines predefined fixed tokens to represent structural elements and low-cardinality categorical features, with a learned subword vocabulary using Byte-Pair Encoding (BPE) for high-cardinality and continuous values. We demonstrate the efficacy of this technique by applying it to a large-scale NetFlow dataset (CIDDS-001), preparing a corpus for a Network Intrusion Detection System (NIDS) foundation model. The evaluation shows that our method is highly efficient, processing over 31 million network flows in under five hours and achieving a significant data compression ratio of 6.18:1. This process resulted in a computationally manageable corpus of over one billion tokens, establishing a viable and generalisable pathway for training foundation models on structured data.