{"list":"other","total":52,"rows":[{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"1909.08053","title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","created":"2019-09-17","cited_by_count":1868},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"2301.05217","title":"Progress measures for grokking via mechanistic interpretability","created":"2023-01-12","cited_by_count":839},{"id":"1909.00512","title":"How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings","created":"2019-09-02","cited_by_count":784},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"2402.00159","title":"Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research","created":"2024-01-31","cited_by_count":483},{"id":"2103.03404","title":"Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth","created":"2021-03-05","cited_by_count":419},{"id":"2502.02013","title":"Layer by Layer: Uncovering Hidden Representations in Language Models","created":"2025-02-04","cited_by_count":311},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1907.12009","title":"Representation Degeneration Problem in Training Natural Language Generation Models","created":"2019-07-28","cited_by_count":255},{"id":"2107.04649","title":"Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization","created":"2021-07-09","cited_by_count":249},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161},{"id":"2401.00448","title":"Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws","created":"2023-12-31","cited_by_count":159},{"id":"2109.04404","title":"All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality","created":"2021-09-09","cited_by_count":156},{"id":"2310.06786","title":"OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text","created":"2023-10-10","cited_by_count":155},{"id":"2206.03126","title":"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse","created":"2022-06-07","cited_by_count":141},{"id":"2310.20707","title":"What's In My Big Data?","created":"2023-10-31","cited_by_count":129},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2403.08540","title":"Language models scale reliably with over-training and on downstream tasks","created":"2024-03-13","cited_by_count":113},{"id":"2302.00294","title":"The geometry of hidden representations of large transformer models","created":"2023-02-01","cited_by_count":112},{"id":"2010.12821","title":"Rethinking embedding coupling in pre-trained language models","created":"2020-10-24","cited_by_count":103},{"id":"2411.04330","title":"Scaling Laws for Precision","created":"2024-11-07","cited_by_count":95},{"id":"2206.13089","title":"Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift","created":"2022-06-27","cited_by_count":94},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"2401.12143","title":"Anisotropy Is Inherent to Self-Attention in Transformers","created":"2024-01-22","cited_by_count":63},{"id":"2503.19206","title":"Overtrained Language Models Are Harder to Fine-Tune","created":"2025-03-24","cited_by_count":58},{"id":"2405.15471","title":"Emergence of a High-Dimensional Abstraction Phase in Language Transformers","created":"2024-05-24","cited_by_count":56},{"id":"2312.10523","title":"Paloma: A Benchmark for Evaluating Language Model Fit","created":"2023-12-16","cited_by_count":54},{"id":"2108.07344","title":"IsoScore: Measuring the Uniformity of Embedding Space Utilization","created":"2021-08-16","cited_by_count":53},{"id":"2305.17212","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","created":"2023-05-26","cited_by_count":53},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41},{"id":"2403.09539","title":"Logits of API-Protected LLMs Leak Proprietary Information","created":"2024-03-14","cited_by_count":40},{"id":"2504.11393","title":"DataDecide: How to Predict Best Pretraining Data with Small Experiments","created":"2025-04-15","cited_by_count":40},{"id":"2311.05928","title":"The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models","created":"2023-11-10","cited_by_count":39},{"id":"2406.03445","title":"Pre-trained Large Language Models Use Fourier Features to Compute Addition","created":"2024-06-05","cited_by_count":39},{"id":"2503.09543","title":"PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs","created":"2025-03-12","cited_by_count":35},{"id":"2509.23024","title":"Tracing the Representation Geometry of Language Models from Pretraining to Post-training","created":"2025-09-27","cited_by_count":35},{"id":"2408.11804","title":"Approaching Deep Learning through the Spectral Dynamics of Weights","created":"2024-08-21","cited_by_count":31},{"id":"2109.03127","title":"Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings","created":"2021-09-07","cited_by_count":27},{"id":"2408.15417","title":"Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations","created":"2024-08-27","cited_by_count":24},{"id":"2404.07647","title":"Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck","created":"2024-04-11","cited_by_count":21},{"id":"2507.13338","title":"Training Transformers with Enforced Lipschitz Constants","created":"2025-07-17","cited_by_count":20},{"id":"2410.11462","title":"Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing","created":"2024-10-15","cited_by_count":10},{"id":"2410.11451","title":"Tending Towards Stability: Convergence Challenges in Small Language Models","created":"2024-10-15","cited_by_count":7},{"id":"2402.03191","title":"Isotropy, Clusters, and Classifiers","created":"2024-02-05","cited_by_count":4},{"id":"2502.08441","title":"Better Embeddings with Coupled Adam","created":"2025-02-12","cited_by_count":4},{"id":"2510.14086","title":"Every Language Model Has a Forgery-Resistant Signature","created":"2025-10-15","cited_by_count":4}],"next_cursor":null}