mlsanityData to 14 Oct 2026

xVal: A Continuous Numerical Tokenization for Scientific Language Models

arXiv 2310.02989 ↗First posted 4 Oct 2023stat.ML cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Oct 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 5; 2026: 9.

27 papers.

Sort:

By area:

  1. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  2. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  3. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  4. 2023ChatPose: Chatting about 3D Human Posecited by 72
  5. 2024Large Language Models as Markov Chainscited by 35
  6. 2024Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenizationcited by 27
  7. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  8. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  9. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 13
  10. 2023Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Datacited by 9
  11. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 8
  12. 2024Interleaving Text and Number Embeddings to Solve Mathemathics Problemscited by 5
15 more
  1. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  2. 2024Can LLMs predict the convergence of Stochastic Gradient Descent?cited by 2
  3. 2026SymTorch: Symbolic Distillation of Neural Networkscited by 2
  4. 2025Generating particle physics Lagrangians with transformerscited by 1
  5. 2025multivariateGPT: a decoder-only transformer for multivariate categorical and numeric datacited by 1
  6. 2026Enhancing Numerical Prediction in LLMs via Smooth MMD Alignmentcited by 1
  7. 2024On Languaging a Simulation Enginecited by 0
  8. 2025BiomechGPT: Extending Motion-Language Models to Clinical Motion Understandingcited by 0
  9. 2026From Tokens to Numbers: Continuous Number Modeling for SVG Generationcited by 0
  10. 2026Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenizationcited by 0
  11. 2026Large Language Models as Amortized Pareto-Front Generators for Constrained Bi-Objective Convex Optimizationcited by 0
  12. 2026Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Modelscited by 0
  13. 2026FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matchingcited by 0
  14. 2026ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoningcited by 0
  15. 2026MoMaTune: Model, Match, Then Tune Language Models for Numerical Recordscited by 0