mlsanityData to 14 Oct 2026

Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

arXiv 2604.09585 ↗First posted 27 Feb 2026cs.HC cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 43 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

14 of 43 references.

References · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  4. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  5. 2023Penetrative AI: Making LLMs Comprehend the Physical Worldcited by 64
  6. 2024HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?cited by 54
  7. 2024SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognitioncited by 38
  8. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  9. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  10. 2024By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Promptingcited by 12
  11. 2024Large Language Model-Guided Semantic Alignment for Human Activity Recognitioncited by 6
  12. 2024Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLMcited by 5
2 more
  1. 2025Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understandingcited by 5
  2. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance
  2. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
  3. 2023TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition
  4. 2025GazeLLM: Multimodal LLMs incorporating Human Visual Attention
  5. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  6. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
  7. 2026Knowledge Distillation for LLM-Based Human Activity Recognition in Homes
  8. 2025Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning