LLM Architecture, Scaling Laws, and Economics: A Quick Summary
Not on the map. Why?
Abstract (arXiv)
The current standard architecture of Large Language Models (LLMs) with QKV self-attention is briefly summarized, including the architecture of a typical Transformer. Scaling laws for compute (flops) and memory (parameters plus data) are given, along with their present (2025) rough cost estimates for the parameters of present LLMs of various scales, including discussion of whether DeepSeek should be viewed as a special case. Nothing here is new, but this material seems not otherwise readily available in summary form.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 11 Sep 2025.
Builds on
8 of 14 references.
References · 8
- 2017Attention Is All You Needcited by 60,361
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
- 2014Neural Turing Machinescited by 644
- 2024The rising costs of training frontier AI modelscited by 95
Built on by
Not cited yet.