mlsanityData to 7 Oct 2026

Parameter Golf: What Really Works?

arXiv 2607.01517 ↗First posted 1 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

How far can a language model improve under a strict artifact budget? Parameter Golf posed this question as an open community challenge in which participants trained the best language model, with the complete artifact (training code + compressed weights) required to fit within 16 MB and to be trained in under ten minutes on 8xH100 SXM GPUs. Quality was measured in bits-per-byte (BPB), the average number of bits required to encode each byte of unseen text. We analyze 2,037 pull requests and 1,430 clean-scored submissions from the contest, build a taxonomy of 84 optimization techniques, and measure each technique's contribution to BPB. The verified leaderboard score dropped from 1.2244 to 1.058 BPB across three phases, a 13.6% reduction, despite individual techniques rarely improving BPB by more than 1%. We show that most techniques' gains shrink when re-measured among competitive submissions, isolating the few methods that help regardless of the surrounding stack. Code and data are available at https://github.com/PMP56/pmgolf-analysis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

22 of 37 references.

References · 22

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
10 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2018Universal Transformerscited by 613
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  7. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  8. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  9. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  10. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 39

Built on by

Not cited yet.