# Exploring and Characterizing Large Language Models For Embedded System Development and Debugging

arXiv: https://arxiv.org/abs/2307.03817 · HTML: https://mlsanity.com/paper/2307.03817/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 73 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [RobustFill: Neural Program Learning under Noisy I/O](https://mlsanity.com/paper/1703.07469/) (2017)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)

## Built on by (most cited first, 20 of 20)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions](https://mlsanity.com/paper/2505.14668/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions](https://mlsanity.com/paper/2412.04036/) (2024)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [EmbedGenius: Towards Automated Software Development for Generic Embedded IoT Systems](https://mlsanity.com/paper/2412.09058/) (2024)
- [EmbedAgent: Benchmarking Large Language Models in Embedded System Development](https://mlsanity.com/paper/2506.11003/) (2025)
- [InCoder-32B-Thinking: Industrial Code World Model for Thinking](https://mlsanity.com/paper/2604.03144/) (2026)
- [Consolidating TinyML Lifecycle with Large Language Models: Reality, Illusion, or Opportunity?](https://mlsanity.com/paper/2501.12420/) (2025)
- [Securing LLM-Generated Embedded Firmware through AI Agent-Driven Validation and Patching](https://mlsanity.com/paper/2509.09970/) (2025)
- [ProtocolLLM: RTL Benchmark for SystemVerilog Generation of Communication Protocols](https://mlsanity.com/paper/2506.07945/) (2025)
- [Advances in Large Language Models for Medicine](https://mlsanity.com/paper/2509.18690/) (2025)
- [VoCopilot: Voice-Activated Tracking of Everyday Interactions](https://mlsanity.com/paper/2312.10265/) (2023)
- [CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation](https://mlsanity.com/paper/2507.19904/) (2025)
- [AutoBridge: Automating Smart Device Integration with Centralized Platform](https://mlsanity.com/paper/2507.23178/) (2025)
- [Smarter, not Bigger: Fine-Tuned RAG-Enhanced LLMs for Automotive HIL Testing](https://mlsanity.com/paper/2511.22584/) (2025)
- [Artificial Intelligence as a Catalyst for Innovation in Software Engineering](https://mlsanity.com/paper/2603.10994/) (2026)
- [Embedded Arena: Iterative Optimization via Hardware Feedback](https://mlsanity.com/paper/2606.16190/) (2026)
- [Surveying GenAI-based Automation in Printed Circuit Board Design and Test](https://mlsanity.com/paper/2606.17074/) (2026)
- [Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Development](https://mlsanity.com/paper/2609.30863/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable abilities to generate code, however their ability to develop software for embedded systems, which requires cross-domain knowledge of hardware and software has not been studied. In this paper we develop an extensible, open source hardware-in-the-loop framework to systematically evaluate leading LLMs (GPT-3.5, GPT-4, PaLM 2) to assess their capabilities and limitations for embedded system development. We observe through our study that even when these tools fail to produce working code, they consistently generate helpful reasoning about embedded design tasks. We leverage this finding to study how human programmers interact with these tools, and develop an human-AI based software engineering workflow for building embedded systems. Our evaluation platform for verifying LLM generated programs uses sensor actuator pairs for physical evaluation. We compare all three models with N=450 experiments and find surprisingly that GPT-4 especially shows an exceptional level of cross-domain understanding and reasoning, in some cases generating fully correct programs from a single prompt. In N=50 trials, GPT-4 produces functional I2C interfaces 66% of the time. GPT-4 also produces register-level drivers, code for LoRa communication, and context-specific power optimizations for an nRF52 program resulting in over 740x current reduction to 12.2uA. We also characterize the models' limitations to develop a generalizable human-AI workflow for using LLMs in embedded system development. We evaluate our workflow with 15 users including novice and expert programmers. We find that our workflow improves productivity for all users and increases the success rate for building a LoRa environmental sensor from 25% to 100%, including for users with zero hardware or C/C++ experience.