# Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition

arXiv: https://arxiv.org/abs/2509.10729 · HTML: https://mlsanity.com/paper/2509.10729/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals](https://mlsanity.com/paper/2409.00101/) (2024)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [SensorLM: Learning the Language of Wearable Sensors](https://mlsanity.com/paper/2506.09108/) (2025)
- [Towards Time Series Reasoning with LLMs](https://mlsanity.com/paper/2409.11376/) (2024)
- [Improving Reasoning Performance in Large Language Models via Representation Engineering](https://mlsanity.com/paper/2504.19483/) (2025)
- [DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search](https://mlsanity.com/paper/2410.03864/) (2024)
- [LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data](https://mlsanity.com/paper/2406.14498/) (2024)
- [DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2505.02123/) (2025)
- [Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them](https://mlsanity.com/paper/2408.12023/) (2024)
- [Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors](https://mlsanity.com/paper/2410.20034/) (2024)
- [A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects](https://mlsanity.com/paper/2506.19769/) (2025)

## Built on by (most cited first, 5 of 5)

- [LLMs can construct powerful representations and streamline sample-efficient supervised learning](https://mlsanity.com/paper/2603.11679/) (2026)
- [Front-End Ethics for Sensor-Fused Health Conversational Agents: An Ethical Design Space for Biometrics](https://mlsanity.com/paper/2604.06203/) (2026)
- [TRACE: Temporal Reasoning over Context and Evidence for Activity Recognition in Smart Homes](https://mlsanity.com/paper/2605.02841/) (2026)
- [SensingAgents: A Multi-Agent Collaborative Framework for Robust IMU Activity Recognition](https://mlsanity.com/paper/2605.04608/) (2026)
- [Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating](https://mlsanity.com/paper/2607.23037/) (2026)

## Abstract (arXiv)

Sensor data streams provide valuable information around activities and context for downstream applications, though integrating complementary information can be challenging. We show that large language models (LLMs) can be used for late fusion for activity classification from audio and motion time series data. We curated a subset of data for diverse activity recognition across contexts (e.g., household activities, sports) from the Ego4D dataset. Evaluated LLMs achieved 12-class zero- and one-shot classification F1-scores significantly above chance, with no task-specific training. Zero-shot classification via LLM-based fusion from modality-specific models can enable multimodal temporal applications where there is limited aligned training data for learning a shared embedding space. Additionally, LLM-based fusion can enable model deploying without requiring additional memory and computation for targeted application-specific multimodal models.