mlsanityData to 14 Oct 2026

The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models

arXiv 2309.06236 ↗First posted 12 Sep 2023cs.LG cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

35 of 57 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
23 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  5. 2022Galactica: A Large Language Model for Sciencecited by 744
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  9. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  10. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  11. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  12. 2018Adversarial Reprogramming of Neural Networkscited by 116
  13. 2019Multi-task Self-Supervised Learning for Human Activity Detectioncited by 102
  14. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  15. 2022Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Datacited by 75
  16. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 65
  17. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
  18. 2023Multimodal LLMs for health grounded in individual-specific datacited by 49
  19. 2022IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Textcited by 37
  20. 2021SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Datacited by 31
  21. 2020Self-supervised transfer learning of physiological representations from free-living wearable datacited by 27
  22. 2021Multilingual Translation via Grafting Pre-trained Language Modelscited by 10
  23. 2021Generative Pre-Trained Transformer for Cardiac Abnormality Detectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 16; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  2. 2024Large Language Models for Time Series: A Surveycited by 114
  3. 2024PaPaGei: Open Foundation Models for Optical Physiological Signalscited by 69
  4. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  5. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  6. 2024Position: What Can Large Language Models Tell Us about Time Series Analysiscited by 47
  7. 2024SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognitioncited by 38
  8. 2024TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysiscited by 29
  9. 2024Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learningcited by 25
  10. 2024Empowering Time Series Analysis with Foundation Models: A Comprehensive Surveycited by 23
  11. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  12. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
22 more
  1. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  2. 2025When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inferencecited by 12
  3. 2025Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Promptingcited by 7
  4. 2025Efficient numeracy in language models through single-token number embeddingscited by 7
  5. 2025RAG-HAR: Retrieval Augmented Generation-based Human Activity Recognitioncited by 7
  6. 2025BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecastingcited by 6
  7. 2025Eliciting Chain-of-Thought Reasoning for Time Series Analysis using Reinforcement Learningcited by 5
  8. 2025QuAnTS: Question Answering on Time Seriescited by 5
  9. 2024LLM-ABBA: Understanding time series via symbolic approximationcited by 4
  10. 2025TsLLM: Augmenting LLMs for General Time Series Understanding and Predictioncited by 4
  11. 2024Graph-Augmented LLMs for Personalized Health Insights: A Case Study in Sleep Analysiscited by 3
  12. 2025SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Modelcited by 3
  13. 2025Towards Interpretable Time Series Foundation Modelscited by 3
  14. 2026LLM-Assisted Logic Rule Learning: Scaling Human Expertise for Time Series Anomaly Detectioncited by 2
  15. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  16. 2025Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detectioncited by 1
  17. 2025multivariateGPT: a decoder-only transformer for multivariate categorical and numeric datacited by 1
  18. 2025A novel language model for predicting serious adverse event results in clinical trials from their prospective registrationscited by 1
  19. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  20. 2026Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognitioncited by 0
  21. 2026Autoregressive EHR Foundation Models with Multimodal Inputscited by 0
  22. 2026Can Large Language Models Explain Flight Safety Events? A Prior-Guided Semantic LLM-based Approachcited by 0