mlsanityData to 14 Oct 2026

SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition

arXiv 2410.10624 ↗First posted 14 Oct 2024cs.CLcited by 38

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce SensorLLM, a two-stage framework that enables Large Language Models (LLMs) to perform human activity recognition (HAR) from sensor time-series data. Despite their strong reasoning and generalization capabilities, LLMs remain underutilized for motion sensor data due to the lack of semantic context in time-series, computational constraints, and challenges in processing numerical inputs. SensorLLM addresses these limitations through a Sensor-Language Alignment stage, where the model aligns sensor inputs with trend descriptions. Special tokens are introduced to mark channel boundaries. This alignment enables LLMs to capture numerical variations, channel-specific features, and data of varying durations, without requiring human annotations. In the subsequent Task-Aware Tuning stage, we refine the model for HAR classification, achieving performance that matches or surpasses state-of-the-art methods. Our results demonstrate that SensorLLM evolves into an effective sensor learner, reasoner, and classifier through human-intuitive Sensor-Language Alignment, generalizing across diverse HAR datasets. We believe this work establishes a foundation for future research on time-series and text alignment, paving the way for foundation models in sensor data analysis. Our codes are available at https://github.com/zechenli03/SensorLLM.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 76 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

40 of 76 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  9. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  10. 2023A Survey on Multimodal Large Language Modelscited by 909
  11. 2024Chronos: Learning the Language of Time Seriescited by 759
  12. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
28 more
  1. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  3. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  4. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  5. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  6. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  8. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  9. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 172
  10. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  11. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  12. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  13. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 129
  14. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  15. 2024Large Language Models for Time Series: A Surveycited by 114
  16. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  17. 2016Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearablescited by 94
  18. 2017Ensembles of Deep LSTM Learners for Activity Recognition using Wearablescited by 62
  19. 2024HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?cited by 54
  20. 2018On Attention Models for Human Activity Recognitioncited by 34
  21. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  22. 2023ETP: Learning Transferable ECG Representations via ECG-Text Pre-trainingcited by 29
  23. 2023TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognitioncited by 25
  24. 2023CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Maskingcited by 19
  25. 2024By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Promptingcited by 12
  26. 2023Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPTcited by 11
  27. 2024Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Themcited by 10
  28. 2018PerceptionNet: A Deep Convolutional Neural Network for Late Sensor Fusioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 22.

38 papers.

Sort:

By area:

  1. 2025OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Datacited by 36
  2. 2025COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognitioncited by 9
  3. 2025RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequencecited by 7
  4. 2025RAG-HAR: Retrieval Augmented Generation-based Human Activity Recognitioncited by 7
  5. 2026GlucoFM: A Dual-Stream Foundation Model for Continuous Glucose Monitoringcited by 7
  6. 2026CGM-JEPA: Learning Consistent Continuous Glucose Monitor Representations via Predictive Self-Supervised Pretrainingcited by 6
  7. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  8. 2025QuAnTS: Question Answering on Time Seriescited by 5
  9. 2025Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokenscited by 4
  10. 2026RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skillscited by 4
  11. 2025EgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensorcited by 3
  12. 2026Wearable Foundation Models Should Go Beyond Static Encoderscited by 3
26 more
  1. 2025mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMUcited by 2
  2. 2026RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deploymentcited by 2
  3. 2025From Motion Signals to Insights: A Unified Framework for Student Behavior Analysis and Feedback in Physical Education Classescited by 1
  4. 2025Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detectioncited by 1
  5. 2025On-device Large Multi-modal Agent for Human Activity Recognitioncited by 1
  6. 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
  7. 2026A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systemscited by 1
  8. 2026AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wildcited by 1
  9. 2025From Time Series Analysis to Question Answering: A Survey in the LLM Eracited by 0
  10. 2025XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edgecited by 0
  11. 2025SitLLM: Large Language Models for Sitting Posture Health Understanding via Pressure Sensor Datacited by 0
  12. 2026MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signalscited by 0
  13. 2026Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understandingcited by 0
  14. 2026SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Modelscited by 0
  15. 2026Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognitioncited by 0
  16. 2026Towards Green Wearable Computing: A Physics-Aware Spiking Neural Network for Energy-Efficient IMU-based Human Activity Recognitioncited by 0
  17. 2026PULSE: Agentic Investigation with Passive Sensing for Proactive Affective Intervention in Cancer Survivorshipcited by 0
  18. 2026Gravity-Aware Hierarchical Routing for Lightweight SensorLLM on Human Activity Recognitioncited by 0
  19. 2026Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Datacited by 0
  20. 2026Enabling Cloud-Level Accuracy in Edge AI through IoT Data Preprocessingcited by 0
  21. 2026Physical activities enable scalable foundation modelling for broad-spectrum health predictioncited by 0
  22. 2026When More Modalities Hurt: Modality Dropout for Heavy-Duty Vehicle Engine Diagnosticscited by 0
  23. 2026PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignmentcited by 0
  24. 2026OntoAR-HAR: Ontology-Guided Agentic RAG with Personalized Memory for Human Activity Recognitioncited by 0
  25. 2026CHEAR: Confusion-Hierarchy Evidence-Aware Reasoning for Cost-Efficient Human Activity Recognitioncited by 0
  26. 2026Lotus: Learning to Understand Time-series by Learning to Forecastcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model-Guided Semantic Alignment for Human Activity Recognition
  2. 2026Knowledge Distillation for LLM-Based Human Activity Recognition in Homes
  3. 2025MuRAL: A Multi-Resident Ambient Sensor Dataset Annotated with Natural Language for Activities of Daily Living
  4. 2025ZARA: Training-Free Motion Time-Series Reasoning via Evidence-Grounded LLM Agents
  5. 2024LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data
  6. 2023GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning
  7. 2026TRACE-TS: Attribution-Grounded and Traceable Sensor-Language Reasoning for Human Activity Understanding
  8. 2026TeamLLM: Exploring the Capabilities of LLMs for Multimodal Group Interaction Prediction