mlsanityData to 14 Oct 2026

Innovative tokenisation of structured data for LLM training

arXiv 2508.01685 ↗First posted 3 Aug 2025cs.LG cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

Data representation remains a fundamental challenge in machine learning, particularly when adapting sequence-based architectures like Transformers and Large Language Models (LLMs) for structured tabular data. Existing methods often fail to cohesively encode the mix of numerical and categorical features or preserve the inherent structure of tables. This paper introduces a novel, hybrid tokenisation methodology designed to convert tabular data into a unified, sequential format suitable for LLM training. Our approach combines predefined fixed tokens to represent structural elements and low-cardinality categorical features, with a learned subword vocabulary using Byte-Pair Encoding (BPE) for high-cardinality and continuous values. We demonstrate the efficacy of this technique by applying it to a large-scale NetFlow dataset (CIDDS-001), preparing a corpus for a Network Intrusion Detection System (NIDS) foundation model. The evaluation shows that our method is highly efficient, processing over 31 million network flows in under five hours and achieving a significant data compression ratio of 6.18:1. This process resulted in a computationally manageable corpus of over one billion tokens, establishing a viable and generalisable pathway for training foundation models on structured data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 3 Aug 2025.

Builds on

24 of 37 references.

References · 24

  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  4. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  5. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 137
  6. 2024Tokenization Is More Than Compressioncited by 89
  7. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 77
  8. 2024Large Language Model for Table Processing: A Surveycited by 52
  9. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  10. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  11. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  12. 2020A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Modelscited by 30
12 more
  1. 2016An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learningcited by 29
  2. 2025A Comprehensive Survey of Synthetic Tabular Data Generationcited by 28
  3. 2025Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End Systemcited by 23
  4. 2023Curriculum Graph Machine Learning: A Surveycited by 15
  5. 2021Evaluating Various Tokenizers for Arabic Text Classificationcited by 14
  6. 2025LLM Embeddings for Deep Learning on Tabular Datacited by 12
  7. 2025Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directionscited by 9
  8. 2024Tabular Embedding Model (TEM): Finetuning Embedding Models For Tabular RAG Applicationscited by 8
  9. 2024Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifierscited by 6
  10. 2022A Survey of Text Representation Methods and Their Genealogycited by 5
  11. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  12. 2024Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)cited by 2

Built on by

Not cited yet.