{"total":73,"rows":[{"id":"2412.13663","title":"Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference","created":"2024-12-18","cited_by_count":554,"area":null},{"id":"2111.01243","title":"Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey","created":"2021-11-01","cited_by_count":463,"area":null},{"id":"2302.03169","title":"Data Selection for Language Models via Importance Resampling","created":"2023-02-06","cited_by_count":329,"area":null},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265,"area":null},{"id":"2202.08005","title":"Should You Mask 15% in Masked Language Modeling?","created":"2022-02-16","cited_by_count":118,"area":null},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104,"area":null},{"id":"2504.08165","title":"Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora","created":"2025-04-10","cited_by_count":88,"area":null},{"id":"2210.14199","title":"Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models","created":"2022-10-25","cited_by_count":84,"area":null},{"id":"2212.14034","title":"Cramming: Training a Language Model on a Single GPU in One Day","created":"2022-12-28","cited_by_count":81,"area":null},{"id":"2304.08442","title":"The MiniPile Challenge for Data-Efficient Language Models","created":"2023-04-17","cited_by_count":75,"area":null},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52,"area":null},{"id":"2212.10544","title":"Pretraining Without Attention","created":"2022-12-20","cited_by_count":51,"area":null},{"id":"2303.08117","title":"Do Transformers Parse while Predicting the Masked Word?","created":"2023-03-14","cited_by_count":48,"area":null},{"id":"2307.06440","title":"No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models","created":"2023-07-12","cited_by_count":41,"area":null},{"id":"2306.07377","title":"Lost in Translation: Large Language Models in Non-English Content Analysis","created":"2023-06-12","cited_by_count":35,"area":null},{"id":"2312.17482","title":"MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining","created":"2023-12-29","cited_by_count":34,"area":null},{"id":"2108.02170","title":"Curriculum learning for language modeling","created":"2021-08-04","cited_by_count":32,"area":null},{"id":"2108.11193","title":"Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens","created":"2021-08-25","cited_by_count":27,"area":null},{"id":"2109.04711","title":"Pre-train or Annotate? Domain Adaptation with a Constrained Budget","created":"2021-09-10","cited_by_count":27,"area":null},{"id":"2304.03589","title":"On Efficient Training of Large-Scale Deep Learning Models: A Literature Review","created":"2023-04-07","cited_by_count":26,"area":null},{"id":"2305.02538","title":"Cuttlefish: Low-Rank Model Training without All the Tuning","created":"2023-05-04","cited_by_count":25,"area":null},{"id":"2410.19637","title":"A distributional simplicity bias in the learning dynamics of transformers","created":"2024-10-25","cited_by_count":24,"area":null},{"id":"2204.06644","title":"METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals","created":"2022-04-13","cited_by_count":20,"area":null},{"id":"2311.08886","title":"CLIMB: Curriculum Learning for Infant-inspired Model Building","created":"2023-11-15","cited_by_count":20,"area":null},{"id":"2401.02452","title":"The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?","created":"2024-01-04","cited_by_count":20,"area":null},{"id":"2203.01178","title":"DCT-Former: Efficient Self-Attention with Discrete Cosine Transform","created":"2022-03-02","cited_by_count":12,"area":null},{"id":"2301.02654","title":"Does compressing activations help model parallel training?","created":"2023-01-06","cited_by_count":12,"area":null},{"id":"2502.15814","title":"Slamming: Training a Speech Language Model on One GPU in a Day","created":"2025-02-19","cited_by_count":12,"area":null},{"id":"2210.07095","title":"Incorporating Context into Subword Vocabularies","created":"2022-10-13","cited_by_count":11,"area":null},{"id":"2302.06117","title":"The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment","created":"2023-02-13","cited_by_count":11,"area":null},{"id":"2305.01645","title":"Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models","created":"2023-05-02","cited_by_count":11,"area":null},{"id":"2306.17649","title":"Biomedical Language Models are Robust to Sub-optimal Tokenization","created":"2023-06-30","cited_by_count":9,"area":null},{"id":"2312.14567","title":"Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise","created":"2023-12-22","cited_by_count":9,"area":null},{"id":"2306.01497","title":"Data-Efficient French Language Modeling with CamemBERTa","created":"2023-06-02","cited_by_count":8,"area":null},{"id":"2211.04898","title":"Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token","created":"2022-11-09","cited_by_count":7,"area":null},{"id":"2211.17201","title":"ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT","created":"2022-11-30","cited_by_count":7,"area":null},{"id":"2410.24140","title":"Don't Touch My Diacritics","created":"2024-10-31","cited_by_count":7,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2211.05187","title":"Training a Vision Transformer from scratch in less than 24 hours with 1 GPU","created":"2022-11-09","cited_by_count":6,"area":null},{"id":"2402.18508","title":"Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling","created":"2024-02-28","cited_by_count":6,"area":null},{"id":"2410.23261","title":"$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources","created":"2024-10-30","cited_by_count":6,"area":null},{"id":"2208.14111","title":"Transformers with Learnable Activation Functions","created":"2022-08-30","cited_by_count":5,"area":null},{"id":"2306.16900","title":"Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research","created":"2023-06-29","cited_by_count":5,"area":null},{"id":"2310.14423","title":"A Quadratic Synchronization Rule for Distributed Deep Learning","created":"2023-10-22","cited_by_count":5,"area":null},{"id":"2311.00658","title":"Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew","created":"2023-11-01","cited_by_count":5,"area":null},{"id":"2402.13723","title":"The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learning","created":"2024-02-21","cited_by_count":5,"area":null},{"id":"2205.11588","title":"Simple Recurrence Improves Masked Language Models","created":"2022-05-23","cited_by_count":4,"area":null},{"id":"2301.07006","title":"Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks","created":"2023-01-17","cited_by_count":4,"area":null},{"id":"2409.12175","title":"Expanding Expressivity in Transformer Models with M\\\"obiusAttention","created":"2024-09-08","cited_by_count":4,"area":null},{"id":"2502.16249","title":"Linear Attention for Efficient Bidirectional Sequence Modeling","created":"2025-02-22","cited_by_count":4,"area":null},{"id":"2205.14014","title":"What Dense Graph Do You Need for Self-Attention?","created":"2022-05-27","cited_by_count":3,"area":null},{"id":"2210.13536","title":"Effective Pre-Training Objectives for Transformer-based Autoencoders","created":"2022-10-24","cited_by_count":3,"area":null},{"id":"2301.04761","title":"NarrowBERT: Accelerating Masked Language Model Pretraining and Inference","created":"2023-01-11","cited_by_count":3,"area":null},{"id":"2411.05527","title":"How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP","created":"2024-11-08","cited_by_count":3,"area":null},{"id":"2309.08708","title":"Vocabulary-level Memory Efficiency for Language Model Fine-tuning","created":"2023-09-15","cited_by_count":2,"area":null},{"id":"2310.05572","title":"A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers","created":"2023-10-09","cited_by_count":2,"area":null},{"id":"2505.15696","title":"MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation","created":"2025-05-21","cited_by_count":2,"area":null},{"id":"2509.14926","title":"Patent Language Model Pretraining with ModernBERT","created":"2025-09-18","cited_by_count":2,"area":null},{"id":"2510.19493","title":"What is the Best Sequence Length for BABYLM?","created":"2025-10-22","cited_by_count":2,"area":null},{"id":"2512.11411","title":"Sliced ReLU attention: Quasi-linear contextual expressivity via sorting","created":"2025-12-12","cited_by_count":2,"area":null},{"id":"2209.03834","title":"Pre-Training a Graph Recurrent Network for Language Representation","created":"2022-09-08","cited_by_count":1,"area":null},{"id":"2402.19204","title":"PeLLE: Encoder-based language models for Brazilian Portuguese based on open data","created":"2024-02-29","cited_by_count":1,"area":null},{"id":"2505.04643","title":"Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation","created":"2025-05-05","cited_by_count":1,"area":null},{"id":"2505.24452","title":"Stepsize anything: A unified learning rate schedule for budgeted-iteration training","created":"2025-05-30","cited_by_count":1,"area":null},{"id":"2601.03270","title":"Advances and Challenges in Semantic Textual Similarity: A Comprehensive Survey","created":"2025-12-19","cited_by_count":1,"area":null},{"id":"2601.05776","title":"One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models","created":"2026-01-09","cited_by_count":1,"area":null},{"id":"2602.11618","title":"How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?","created":"2026-02-12","cited_by_count":1,"area":null},{"id":"2210.07041","title":"Spontaneous Emerging Preference in Two-tower Language Model","created":"2022-10-13","cited_by_count":0,"area":null},{"id":"2210.12022","title":"Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks","created":"2022-10-21","cited_by_count":0,"area":null},{"id":"2211.12677","title":"Word-Level Representation From Bytes For Language Modeling","created":"2022-11-23","cited_by_count":0,"area":null},{"id":"2406.17615","title":"Aligning Programming Language and Natural Language: Exploring Design Choices in Multi-Modal Transformer-Based Embedding for Bug Localization","created":"2024-06-25","cited_by_count":0,"area":null},{"id":"2410.17094","title":"Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization","created":"2024-10-19","cited_by_count":0,"area":null},{"id":"2512.20941","title":"A Multi-fidelity Double-Delta Wing Dataset and Empirical Scaling Laws for GNN-based Aerodynamic Field Surrogate","created":"2025-12-24","cited_by_count":0,"area":null}],"next_cursor":null}