mlsanityData to 14 Oct 2026

Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels

arXiv 2608.25053 ↗First posted 25 Aug 2026cs.AR cs.AI cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and is publicly released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference. Hydra's source code and the collected per-prompt trace corpus are available open-source at: https://github.com/amirtaherin/hydra

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

47 of 71 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
35 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  3. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  5. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  6. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2019MLPerf Inference Benchmarkcited by 199
  9. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  10. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  11. 2023Large Language Models for Robotics: A Surveycited by 133
  12. 2021MLPerf Tiny Benchmarkcited by 124
  13. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  14. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  15. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
  16. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 34
  17. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  18. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  19. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  20. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  21. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  22. 2021DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devicescited by 26
  23. 2023Mobile Foundation Model as Firmwarecited by 25
  24. 2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
  25. 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inferencecited by 22
  26. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  27. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Modelscited by 17
  28. 2025Edge-First Language Model Inference: Models, Metrics, and Tradeoffscited by 13
  29. 2024PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platformscited by 8
  30. 2025Cross-Platform Scaling of Vision-Language-Action Models from Edge to Cloud GPUscited by 8
  31. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  32. 2026Human Cognition in Machines: A Unified Perspective of World Modelscited by 5
  33. 2024Generative AI on the Edge: Architecture and Performance Evaluationcited by 4
  34. 2025LLMPi: Optimizing LLMs for High-Throughput on Raspberry Picited by 3
  35. 2025EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Modelcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Albireo: Adaptive, Energy-Efficient Inference Framework for Video Object Detection on the Edgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  2. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
  3. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  4. 2026LLM Inference on IMC-NoC Architecture with Balanced Dataflow and Fine-Grained Parallelism
  5. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  6. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
  7. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices
  8. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study