mlsanityData to 14 Oct 2026

LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

arXiv 2607.26491 ↗First posted 29 Jul 2026cs.AR cs.AI cs.ETcited by 1

Not on the map. Why?

Abstract (arXiv)

The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs. A key contributor to chip energy dissipation is data movement between limited on-chip cache and off-chip High Bandwidth Memory (HBM). Meanwhile, emerging memory technologies such as monolithic 3D (M3D) integration of cache memories at the Back-End-Of-Line (BEOL) of logic chips enable larger and denser on-chip memories, creating new opportunities to reduce costly off-chip traffic. However, it remains unclear whether continuously scaling on-chip memory using emerging technologies can effectively improve the energy efficiency of LLM serving. To address this gap, we develop LLMET (LLM with Emerging Technology), a validated cross-layer simulation framework, and conduct a comprehensive study on the impact of large-capacity on-chip memory technologies across a broad range of models, applications and platforms. Utilizing M3D technology to expand the L2 cache from 40MB to 1GB yields a 44% reduction in chip energy during the Llama3.1-70B prefill phase with a 16K context window, based on LLMET simulation on a dual NVIDIA A100 GPU setup. On the 8x NVIDIA B200-like platform, extending the L2 cache from 128MB to 4GB saves the prefill energy by up to 24%. For the edge platform and workloads, the decode energy saving reaches 30% when increasing the 8MB cache size to 256MB. These results highlight the promise of ultra-large on-chip memories for energy-efficient LLM serving systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

20 of 42 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
8 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  5. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  6. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6
  7. 2025CMOS+X: Stacking Persistent Embedded Memories based on Oxide Transistors upon GPGPU Platformscited by 2
  8. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memoriescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hardware Acceleration of Block-Diffusion LLM for Edge Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
  2. 2026A CXL Memory Rack for Multi-Turn LLM Serving
  3. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator
  4. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash
  5. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
  6. 2024Towards Sustainable Large Language Model Serving
  7. 2025Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving
  8. 2026Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware