# An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding

arXiv: https://arxiv.org/abs/2505.01743 · HTML: https://mlsanity.com/paper/2505.01743/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 60 references that are in the map.
- First posted on arXiv 3 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles](https://mlsanity.com/paper/2104.00946/) (2021)
- [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows](https://mlsanity.com/paper/2402.10379/) (2024)
- [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](https://mlsanity.com/paper/2107.12626/) (2021)
- [On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond](https://mlsanity.com/paper/2206.05187/) (2022)
- [LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces](https://mlsanity.com/paper/2403.19857/) (2024)
- [VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments](https://mlsanity.com/paper/2404.02508/) (2024)
- [SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions](https://mlsanity.com/paper/2412.04036/) (2024)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [DVANet: Disentangling View and Action Features for Multi-View Action Recognition](https://mlsanity.com/paper/2312.05719/) (2023)
- [Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations](https://mlsanity.com/paper/2009.07420/) (2020)
- [3DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Information](https://mlsanity.com/paper/2006.07743/) (2020)
- [LLMs Can Evolve Continually on Modality for X-Modal Reasoning](https://mlsanity.com/paper/2410.20178/) (2024)
- [MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels](https://mlsanity.com/paper/2404.01958/) (2024)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks](https://mlsanity.com/paper/2006.07744/) (2020)
- [A Multi-spectral Dataset for Evaluating Motion Estimation Systems](https://mlsanity.com/paper/2007.00622/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning](https://mlsanity.com/paper/2512.07136/) (2025)

## Abstract (arXiv)

The rapid advancements in Large Vision Language Models (LVLMs) offer the potential to surpass conventional labeling by generating richer, more detailed descriptions of on-device human behavior understanding (HBU) in low-resolution vision systems, such as depth, thermal, and infrared. However, existing large vision language model (LVLM) approaches are unable to understand low-resolution data well as they are primarily designed for high-resolution data, such as RGB images. A quick fixing approach is to caption a large amount of low-resolution data, but it requires a significant amount of labor-intensive annotation efforts. In this paper, we propose a novel, labor-saving system, Llambda, designed to support low-resolution HBU. The core idea is to leverage limited labeled data and a large amount of unlabeled data to guide LLMs in generating informative captions, which can be combined with raw data to effectively fine-tune LVLM models for understanding low-resolution videos in HBU. First, we propose a Contrastive-Oriented Data Labeler, which can capture behavior-relevant information from long, low-resolution videos and generate high-quality pseudo labels for unlabeled data via contrastive learning. Second, we propose a Physical-Knowledge Guided Captioner, which utilizes spatial and temporal consistency checks to mitigate errors in pseudo labels. Therefore, it can improve LLMs' understanding of sequential data and then generate high-quality video captions. Finally, to ensure on-device deployability, we employ LoRA-based efficient fine-tuning to adapt LVLMs for low-resolution data. We evaluate Llambda using a region-scale real-world testbed and three distinct low-resolution datasets, and the experiments show that Llambda outperforms several state-of-the-art LVLM systems up to 40.03% on average Bert-Score.