# Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models

arXiv: https://arxiv.org/abs/2603.23668 · HTML: https://mlsanity.com/paper/2603.23668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 18 of 18)

- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [A Survey on Hardware Accelerators for Large Language Models](https://mlsanity.com/paper/2401.09890/) (2024)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inference](https://mlsanity.com/paper/2110.00478/) (2021)
- [On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Accelerators](https://mlsanity.com/paper/2306.06923/) (2023)
- [HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation](https://mlsanity.com/paper/2105.01585/) (2021)
- [Learned Hardware/Software Co-Design of Neural Accelerators](https://mlsanity.com/paper/2010.02075/) (2020)
- [ATM-Net: Adaptive Termination and Multi-Precision Neural Networks for Energy-Harvested Edge Intelligence](https://mlsanity.com/paper/2502.09822/) (2025)
- [DynaSplit: A Hardware-Software Co-Design Framework for Energy-Aware Inference on Edge](https://mlsanity.com/paper/2410.23881/) (2024)
- [Hardware-Software Co-Design for Accelerating Transformer Inference Leveraging Compute-in-Memory](https://mlsanity.com/paper/2502.12344/) (2025)
- [Forecasting and Visualizing Air Quality from Sky Images with Vision-Language Models](https://mlsanity.com/paper/2509.15076/) (2025)
- [The Energy-Efficient Hierarchical Neural Network with Fast FPGA-Based Incremental Learning](https://mlsanity.com/paper/2509.15097/) (2025)
- [A Digital SRAM-Based Compute-In-Memory Macro for Weight-Stationary Dynamic Matrix Multiplication in Transformer Attention Score Computation](https://mlsanity.com/paper/2511.12152/) (2025)
- [RISC-V Based TinyML Accelerator for Depthwise Separable Convolutions in Edge AI](https://mlsanity.com/paper/2511.21232/) (2025)

## Abstract (arXiv)

The rapid deployment of machine learning across platforms from milliwatt-class TinyML devices to large language models has made energy efficiency a primary constraint for sustainable AI. Across these scales, performance and energy are increasingly limited by data movement and memory-system behavior rather than by arithmetic throughput alone. This work reviews energy efficient software hardware codesign methods spanning edge inference and training to datacenter-scale LLM serving, covering accelerator architectures (e.g., ASIC/FPGA dataflows, processing-/compute-in-memory designs) and system-level techniques (e.g., partitioning, quantization, scheduling, and runtime adaptation). We distill common design levers and trade-offs, and highlight recurring gaps including limited cross-platform generalization, large and costly co-design search spaces, and inconsistent benchmarking across workloads and deployment settings. Finally, we outline a hierarchical decomposition perspective that maps optimization strategies to computational roles and supports incremental adaptation, offering practical guidance for building energy and carbon aware ML systems.