mlsanityData to 7 Oct 2026

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

arXiv 2407.10385 ↗First posted 15 Jul 2024cs.CL cs.AI cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 61 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

25 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  9. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  11. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 444
12 more
  1. 2021True Few-Shot Learning with Language Modelscited by 343
  2. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  3. 2023Large Language Models as General Pattern Machinescited by 196
  4. 2020Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunitiescited by 125
  5. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  6. 2024Large Language Models for Time Series: A Surveycited by 113
  7. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  8. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  9. 2023Penetrative AI: Making LLMs Comprehend the Physical Worldcited by 64
  10. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58
  11. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  12. 2020Sensor-based Continuous Authentication of Smartphones' Users Using Behavioral Biometrics: A Contemporary Surveycited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 3.

12 papers.

  1. 2024SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognitioncited by 35
  2. 2025SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactionscited by 11
  3. 2025Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Promptingcited by 7
  4. 2025An LLM-Powered Agent for Physiological Data Analysis: A Case Study on PPG-based Heart Rate Estimationcited by 4
  5. 2025LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Modelscited by 3
  6. 2026ConSensus: Multi-Agent Collaboration for Multimodal Sensingcited by 3
  7. 2024SensorBench: Benchmarking LLMs in Coding-Based Sensor Processingcited by 2
  8. 2025MobQA: A Benchmark Dataset for Semantic Understanding of Human Mobility Data through Question Answeringcited by 1
  9. 2025SignalLLM: A General-Purpose LLM Agent Framework for Automated Signal Processingcited by 1
  10. 2025Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Modelscited by 0
  11. 2026Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognitioncited by 0
  12. 2026TimeThink: Eliciting Compositional Reasoning in Timeseries Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  2. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  3. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
  4. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
  5. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  6. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
  7. 2023Multimodal Large Language Model for Visual Navigation
  8. 2025Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning