mlsanityData to 7 Oct 2026

Tokenization of Gaze Data

arXiv 2503.22145 ↗First posted 28 Mar 2025cs.LG cs.CL cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

A considerable part of the performance of today's large language models (LLM's) and multimodal large language models (MLLM's) depends on their tokenization strategies. While tokenizers are extensively researched for textual and visual input, there is no research on tokenization strategies for gaze data due to its nature. However, a corresponding tokenization strategy would allow using the vision capabilities of pre-trained MLLM's for gaze data, for example, through fine-tuning. In this paper, we aim to close this research gap by analyzing five different tokenizers for gaze data on three different datasets for the forecasting and generation of gaze data through LLMs (cf.~creffig:teaser). We evaluate the tokenizers regarding their reconstruction and compression abilities. Further, we train an LLM for each tokenization strategy, measuring its generative and predictive performance. Overall, we found that a quantile tokenizer outperforms all others in predicting the gaze positions and k-means is best when predicting gaze velocities.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 60 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

22 of 60 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
10 more
  1. 2020Jukebox: A Generative Model for Musiccited by 535
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  4. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 265
  5. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  6. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  7. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  8. 2020Fast WordPiece Tokenizationcited by 85
  9. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  10. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation
  2. 2023What Makes for Good Visual Tokenizers for Large Language Models?
  3. 2023Planting a SEED of Vision in Large Language Model
  4. 2025TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model
  5. 2026TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
  6. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
  7. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
  8. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming