# LightLLM: A Versatile Large Language Model for Predictive Light Sensing

arXiv: https://arxiv.org/abs/2411.15211 · HTML: https://mlsanity.com/paper/2411.15211/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [NetLLM: Adapting Large Language Models for Networking](https://mlsanity.com/paper/2402.02338/) (2024)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [Deep Photovoltaic Nowcasting](https://mlsanity.com/paper/1810.06327/) (2018)
- [Benchmarking of Deep Learning Irradiance Forecasting Models from Sky Images -- an in-depth Analysis](https://mlsanity.com/paper/2102.00721/) (2021)
- [A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks](https://mlsanity.com/paper/2110.01434/) (2021)
- [SKIPP'D: a SKy Images and Photovoltaic Power Generation Dataset for Short-term Solar Forecasting](https://mlsanity.com/paper/2207.00913/) (2022)
- [Large Model for Small Data: Foundation Model for Cross-Modal RF Human Activity Recognition](https://mlsanity.com/paper/2410.19766/) (2024)
- [SolarGest: Ubiquitous and Battery-free Gesture Recognition using Solar Cells](https://mlsanity.com/paper/1812.01766/) (2018)

## Built on by (most cited first, 5 of 5)

- [RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks](https://mlsanity.com/paper/2608.12004/) (2026)
- [Enabling Near-realtime Remote Sensing via Satellite-Ground Collaboration of Large Vision-Language Models](https://mlsanity.com/paper/2510.24242/) (2025)
- [Adaptive and Robust Data Poisoning Detection and Sanitization in Wearable IoT Systems using Large Language Models](https://mlsanity.com/paper/2511.02894/) (2025)
- [Design and Implementation of Agentic Orchestrations and Orchestration of Agents](https://mlsanity.com/paper/2606.31518/) (2026)
- [TopoBrick: Agentic Topology Sampling of Exogenous Variables for Zero-Shot Building IoT Forecasting](https://mlsanity.com/paper/2607.06349/) (2026)

## Abstract (arXiv)

We propose LightLLM, a model that fine tunes pre-trained large language models (LLMs) for light-based sensing tasks. It integrates a sensor data encoder to extract key features, a contextual prompt to provide environmental information, and a fusion layer to combine these inputs into a unified representation. This combined input is then processed by the pre-trained LLM, which remains frozen while being fine-tuned through the addition of lightweight, trainable components, allowing the model to adapt to new tasks without altering its original parameters. This approach enables flexible adaptation of LLM to specialized light sensing tasks with minimal computational overhead and retraining effort. We have implemented LightLLM for three light sensing tasks: light-based localization, outdoor solar forecasting, and indoor solar estimation. Using real-world experimental datasets, we demonstrate that LightLLM significantly outperforms state-of-the-art methods, achieving 4.4x improvement in localization accuracy and 3.4x improvement in indoor solar estimation when tested in previously unseen environments. We further demonstrate that LightLLM outperforms ChatGPT-4 with direct prompting, highlighting the advantages of LightLLM's specialized architecture for sensor data fusion with textual prompts.