mlsanityData to 7 Oct 2026

Penetrative AI: Making LLMs Comprehend the Physical World

arXiv 2310.09605 ↗First posted 14 Oct 2023cs.AI cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Recent developments in Large Language Models (LLMs) have demonstrated their remarkable capabilities across a range of tasks. Questions, however, persist about the nature of LLMs and their potential to integrate common-sense human knowledge when performing tasks involving information about the real physical world. This paper delves into these questions by exploring how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that we term "Penetrative AI". The paper explores such an extension at two levels of LLMs' ability to penetrate into the physical world via the processing of sensory signals. Our preliminary findings indicate that LLMs, with ChatGPT being the representative example in our exploration, have considerable and unique proficiency in employing the embedded world knowledge for interpreting IoT sensor data and reasoning over them about tasks in the physical realm. Not only this opens up new applications for LLMs beyond traditional text-based tasks, but also enables new ways of incorporating human knowledge in cyber-physical systems.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 29 of its 60 references that are in the map. First posted on arXiv 14 Oct 2023.

Builds on

29 of 60 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
17 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2021MetaICL: Learning to Learn In Contextcited by 450
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  12. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  13. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  14. 2023Large Language Models as General Pattern Machinescited by 196
  15. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 192
  16. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  17. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 26; 2025: 24; 2026: 13.

64 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2023LLMLight: Large Language Models as Traffic Signal Control Agentscited by 73
  3. 2024HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?cited by 52
  4. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  5. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  6. 2024LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Tracescited by 28
  7. 2025A Survey of Personalization: From RAG to Agentcited by 26
  8. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  9. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  10. 2024LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devicescited by 21
  11. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
  12. 2024Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Datacited by 17
38 more
  1. 2025AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applicationscited by 17
  2. 2025GIScience in the Era of Artificial Intelligence: A Research Agenda Towards Autonomous GIScited by 13
  3. 2024By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Promptingcited by 12
  4. 2024When IoT Meet LLMs: Applications and Challengescited by 11
  5. 2025SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactionscited by 11
  6. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10
  7. 2025GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Developmentcited by 10
  8. 2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
  9. 2025Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Promptingcited by 7
  10. 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 7
  11. 2024Enabling On-Device LLMs Personalization with Smartphone Sensingcited by 6
  12. 2024EmbedGenius: Towards Automated Software Development for Generic Embedded IoT Systemscited by 6
  13. 2025Wi-Chat: Large Language Model Powered Wi-Fi Sensingcited by 6
  14. 2024Large Language Model-Guided Semantic Alignment for Human Activity Recognitioncited by 5
  15. 2025Foundation Models for CPS-IoT: Opportunities and Challengescited by 5
  16. 2025Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understandingcited by 5
  17. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  18. 2025DailyLLM: Context-Aware Activity Log Generation Using Multi-Modal Sensors and LLMscited by 5
  19. 2025ChatIoT: Large Language Model-based Security Assistant for Internet of Things with Retrieval-Augmented Generationcited by 4
  20. 2025Are We There Yet? A Measurement Study of Efficiency for LLM Applications on Mobile Devicescited by 4
  21. 2024FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systemscited by 3
  22. 2024CogErgLLM: Exploring Large Language Model Systems Design Perspective Using Cognitive Ergonomicscited by 3
  23. 2025RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Languagecited by 3
  24. 2026ConSensus: Multi-Agent Collaboration for Multimodal Sensingcited by 3
  25. 2024SensorBench: Benchmarking LLMs in Coding-Based Sensor Processingcited by 2
  26. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  27. 2024A Unified Framework for Context-Aware IoT Management and State-of-the-Art IoT Traffic Anomaly Detectioncited by 2
  28. 2025Toward Foundation Models for Online Complex Event Detection in CPS-IoT: A Case Studycited by 2
  29. 2026IoT-Brain: Grounding LLMs for Semantic-Spatial Sensor Schedulingcited by 2
  30. 2024Tapping in a Remote Vehicle's onboard LLM to Complement the Ego Vehicle's Field-of-Viewcited by 1
  31. 2024INVARLLM: LLM-assisted Physical Invariant Extraction for Cyber-Physical Systems Anomaly Detectioncited by 1
  32. 2025Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Modelscited by 1
  33. 2025SignalLLM: A General-Purpose LLM Agent Framework for Automated Signal Processingcited by 1
  34. 2025ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecologycited by 1
  35. 2026UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHRcited by 1
  36. 2026AutORAN: LLM-driven Natural Language Programming for Agile xApp Developmentcited by 1
  37. 2024CoRAST: Towards Foundation Model-Powered Correlated Data Analysis in Resource-Constrained CPS and IoTcited by 0
  38. 2024AutoLife: Automatic Life Journaling with Smartphones and LLMscited by 0

50 most cited of 64 All 64 (JSON)