mlsanityData to 14 Oct 2026

PLDR-LLM: Large Language Model from Power Law Decoder Representations

arXiv 2410.16703 ↗First posted 22 Oct 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We present the Large Language Model from Power Law Decoder Representations (PLDR-LLM), a language model that leverages non-linear and linear transformations through Power Law Graph Attention mechanism to generate well-defined deductive and inductive outputs. We pretrain the PLDR-LLMs of varying layer sizes with a small batch size of 32 and ~8B tokens from the RefinedWeb dataset, and show that they achieve competitive performance in zero-shot and few-shot settings compared to scaled dot-product LLMs of similar model size reported in the literature. We show that deductive outputs of PLDR-LLMs can be used to compare model characteristics or improve the performance by introducing the Directed Acyclic Graph (DAG) loss as a metric and regularizer. Our results indicate that the initial maximum learning rate and warm-up steps have a lasting impact on deductive outputs throughout the pretraining. We provide a detailed description of PLDR-LLM architecture, its implementation and the pretraining procedure.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2016Layer Normalizationcited by 5,669
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
21 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  11. 2018DAGs with NO TEARS: Continuous Optimization for Structure Learningcited by 752
  12. 2020Aligning AI With Shared Human Valuescited by 723
  13. 2016Language Modeling with Gated Convolutional Networkscited by 702
  14. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  15. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  16. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  17. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  18. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  19. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  20. 2019CoulGAT: An Experiment on Interpretability of Graph Attention Networkscited by 6
  21. 2021Power Law Graph Transformer for Machine Translation and Representation Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inferencecited by 3
  2. 2026PLDR-LLMs Reason At Self-Organized Criticalitycited by 2
  3. 2026Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inferencecited by 1
  4. 2026Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reductioncited by 0