# xVal: A Continuous Numerical Tokenization for Scientific Language Models

arXiv: https://arxiv.org/abs/2310.02989 · HTML: https://mlsanity.com/paper/2310.02989/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2023.

## Built on by (most cited first, 27 of 27)

- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [ChatPose: Chatting about 3D Human Pose](https://mlsanity.com/paper/2311.18836/) (2023)
- [Large Language Models as Markov Chains](https://mlsanity.com/paper/2410.02724/) (2024)
- [Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization](https://mlsanity.com/paper/2412.05244/) (2024)
- [Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trials](https://mlsanity.com/paper/2409.04481/) (2024)
- [Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning](https://mlsanity.com/paper/2406.13846/) (2024)
- [Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models](https://mlsanity.com/paper/2411.02083/) (2024)
- [Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data](https://mlsanity.com/paper/2312.11502/) (2023)
- [AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning](https://mlsanity.com/paper/2508.07470/) (2025)
- [Interleaving Text and Number Embeddings to Solve Mathemathics Problems](https://mlsanity.com/paper/2410.19353/) (2024)
- [MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining](https://mlsanity.com/paper/2509.06806/) (2025)
- [Can LLMs predict the convergence of Stochastic Gradient Descent?](https://mlsanity.com/paper/2408.01736/) (2024)
- [SymTorch: Symbolic Distillation of Neural Networks](https://mlsanity.com/paper/2602.21307/) (2026)
- [Generating particle physics Lagrangians with transformers](https://mlsanity.com/paper/2501.09729/) (2025)
- [multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data](https://mlsanity.com/paper/2505.21680/) (2025)
- [Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment](https://mlsanity.com/paper/2606.27731/) (2026)
- [On Languaging a Simulation Engine](https://mlsanity.com/paper/2402.16482/) (2024)
- [BiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding](https://mlsanity.com/paper/2505.18465/) (2025)
- [From Tokens to Numbers: Continuous Number Modeling for SVG Generation](https://mlsanity.com/paper/2602.02820/) (2026)
- [Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization](https://mlsanity.com/paper/2604.16775/) (2026)
- [Large Language Models as Amortized Pareto-Front Generators for Constrained Bi-Objective Convex Optimization](https://mlsanity.com/paper/2605.12106/) (2026)
- [Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models](https://mlsanity.com/paper/2608.30751/) (2026)
- [FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matching](https://mlsanity.com/paper/2609.35673/) (2026)
- [ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning](https://mlsanity.com/paper/2609.39294/) (2026)
- [MoMaTune: Model, Match, Then Tune Language Models for Numerical Records](https://mlsanity.com/paper/2610.15235/) (2026)

## Abstract (arXiv)

Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.