# Toward Sustainability-Aware LLM Inference on Edge Clusters

arXiv: https://arxiv.org/abs/2512.04088 · HTML: https://mlsanity.com/paper/2512.04088/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2025.

## Built on by (most cited first, 3 of 3)

- [CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing](https://mlsanity.com/paper/2603.27420/) (2026)
- [Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study](https://mlsanity.com/paper/2606.21428/) (2026)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)

## Abstract (arXiv)

Large language models (LLMs) require substantial computational resources, leading to significant carbon emissions and operational costs. Although training is energy-intensive, the long-term environmental burden arises from inference, amplified by the massive global query volume. Cloud-based inference offers scalability but suffers from latency and bandwidth constraints due to centralized processing and continuous data transfer. Edge clusters instead can mitigate these limitations by enabling localized execution, yet they face trade-offs between performance, energy efficiency, and device constraints. This short paper presents a sustainability-aware LLM inference for edge clusters comprising NVIDIA Jetson Orin NX (8GB) and Nvidia Ada 2000 (16GB) devices. It aims to balance inference latency and carbon footprint through carbon- and latency-aware routing strategies, guided by empirical benchmarking of energy consumption and execution time across diverse prompts and batch (i.e., group of prompts) configurations. We compared baseline greedy strategies to carbon-aware and latency-aware strategies in prompt routing to specific hardware based on benchmarking information. Experimental evaluation shows that a batch size of four prompts achieves a trade-off between throughput, energy efficiency, while larger batches risk GPU memory saturation.