mlsanityData to 7 Oct 2026

A Generalized Optimization Engine (GOE) for Edge AI Inference Acceleration

arXiv 2608.28652 ↗First posted 19 Aug 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Artificial intelligence (AI) models have demonstrated remarkable capabilities across various domains, yet their widespread deployment is impeded by significant computational costs, particularly on resource-constrained devices. This paper explores the theoretical underpinnings of various AI model optimization techniques, algorithms, and abstractions, discussing their potential to reduce computational complexity, memory footprint, latency, and power consumption. Furthermore, we propose a comprehensive hardware (HW) and model-agnostic generalized optimization architecture that integrates these techniques for improved efficiency. Our study underscores the critical role of such a generalized optimization system in preparing model deployment over resource-constrained heterogeneous hardware in a tactical environment. As a concrete demonstration, we show that GOE-compressed language models deploy and run on a GPU-less edge CPU, and that the choice of compression method, not merely its nominal bit-width, determines whether task accuracy survives deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 19 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

11 of 19 references.

References · 11

  1. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  2. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  3. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  5. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  6. 2021CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deploymentcited by 26
  7. 2025TorchAO: PyTorch-Native Training-to-Serving Model Optimizationcited by 25
  8. 2021Enabling NAS with Automated Super-Network Generationcited by 11
  9. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  10. 2023Edge AI Inference in Heterogeneous Constrained Computing: Feasibility and Opportunitiescited by 2
  11. 2022ESPnet-ONNX: Bridging a Gap Between Research and Productioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  2. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
  3. 2026EdgeFM: Efficient Edge Inference for Vision-Language Models
  4. 2024Inference Performance Optimization for Large Language Models on CPUs
  5. 2026From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
  6. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  7. 2025EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
  8. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment