{"list":"other","total":67,"rows":[{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2303.08774","title":"GPT-4 Technical Report","created":"2023-03-15","cited_by_count":18396},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"1602.05629","title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","created":"2016-02-17","cited_by_count":7669},{"id":"1910.13461","title":"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension","created":"2019-10-29","cited_by_count":5893},{"id":"2204.02311","title":"PaLM: Scaling Language Modeling with Pathways","created":"2022-04-05","cited_by_count":5326},{"id":"1612.00796","title":"Overcoming catastrophic forgetting in neural networks","created":"2016-12-02","cited_by_count":4787},{"id":"1911.02116","title":"Unsupervised Cross-lingual Representation Learning at Scale","created":"2019-11-05","cited_by_count":3935},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1704.05426","title":"A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference","created":"2017-04-18","cited_by_count":2682},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"1808.06226","title":"SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing","created":"2018-08-19","cited_by_count":1989},{"id":"2211.05100","title":"BLOOM: A 176B-Parameter Open-Access Multilingual Language Model","created":"2022-11-09","cited_by_count":1984},{"id":"2010.11934","title":"mT5: A massively multilingual pre-trained text-to-text transformer","created":"2020-10-22","cited_by_count":1818},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"2004.10964","title":"Don't Stop Pretraining: Adapt Language Models to Domains and Tasks","created":"2020-04-23","cited_by_count":1565},{"id":"1910.02054","title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","created":"2019-10-04","cited_by_count":1301},{"id":"2305.10403","title":"PaLM 2 Technical Report","created":"2023-05-17","cited_by_count":1162},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"1901.07291","title":"Cross-lingual Language Model Pretraining","created":"2019-01-22","cited_by_count":1090},{"id":"1803.02999","title":"On First-Order Meta-Learning Algorithms","created":"2018-03-08","cited_by_count":1018},{"id":"1708.00055","title":"SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation","created":"2017-07-31","cited_by_count":1001},{"id":"1704.04683","title":"RACE: Large-scale ReAding Comprehension Dataset From Examinations","created":"2017-04-15","cited_by_count":861},{"id":"2204.06745","title":"GPT-NeoX-20B: An Open-Source Autoregressive Language Model","created":"2022-04-14","cited_by_count":713},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"2304.11277","title":"PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel","created":"2023-04-21","cited_by_count":660},{"id":"1906.01502","title":"How multilingual is Multilingual BERT?","created":"2019-06-04","cited_by_count":645},{"id":"1805.09767","title":"Local SGD Converges Fast and Communicates Little","created":"2018-05-24","cited_by_count":605},{"id":"1910.11856","title":"On the Cross-lingual Transferability of Monolingual Representations","created":"2019-10-25","cited_by_count":578},{"id":"2007.14390","title":"Flower: A Friendly Federated Learning Research Framework","created":"2020-07-28","cited_by_count":455},{"id":"2012.15613","title":"How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models","created":"2020-12-31","cited_by_count":318},{"id":"2402.07827","title":"Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model","created":"2024-02-12","cited_by_count":295},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"2309.11235","title":"OpenChat: Advancing Open-source Language Models with Mixed-Quality Data","created":"2023-09-20","cited_by_count":252},{"id":"2406.15126","title":"On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey","created":"2024-06-14","cited_by_count":244},{"id":"2304.05613","title":"ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning","created":"2023-04-12","cited_by_count":206},{"id":"1808.07217","title":"Don't Use Large Mini-Batches, Use Local SGD","created":"2018-08-22","cited_by_count":202},{"id":"2209.15430","title":"Relative representations enable zero-shot latent space communication","created":"2022-09-30","cited_by_count":191},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"1905.03817","title":"On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization","created":"2019-05-09","cited_by_count":180},{"id":"2311.08105","title":"DiLoCo: Distributed Low-Communication Training of Language Models","created":"2023-11-14","cited_by_count":121},{"id":"2407.13623","title":"Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies","created":"2024-07-18","cited_by_count":118},{"id":"1611.01462","title":"Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling","created":"2016-11-04","cited_by_count":92},{"id":"2006.07264","title":"Low-resource Languages: A Review of Past Work and Future Challenges","created":"2020-06-12","cited_by_count":91},{"id":"2405.17399","title":"Transformers Can Do Arithmetic with the Right Embeddings","created":"2024-05-27","cited_by_count":78},{"id":"2309.06497","title":"A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale","created":"2023-09-12","cited_by_count":77},{"id":"2305.10266","title":"Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability","created":"2023-05-17","cited_by_count":62},{"id":"2010.12777","title":"Improving Multilingual Models with Language-Clustered Vocabularies","created":"2020-10-24","cited_by_count":53},{"id":"2406.03476","title":"Does your data spark joy? Performance gains from domain upsampling at the end of training","created":"2024-06-05","cited_by_count":52},{"id":"2404.09492","title":"Bridging the Gap between Different Vocabularies for LLM Ensemble","created":"2024-04-15","cited_by_count":45},{"id":"2307.01163","title":"Improving Language Plasticity via Pretraining with Active Forgetting","created":"2023-07-03","cited_by_count":43},{"id":"2306.10968","title":"BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models","created":"2023-06-19","cited_by_count":42},{"id":"2405.10936","title":"A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers","created":"2024-05-17","cited_by_count":40},{"id":"2402.12204","title":"Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages","created":"2024-02-19","cited_by_count":36},{"id":"2408.04303","title":"Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP","created":"2024-08-08","cited_by_count":36},{"id":"2311.09205","title":"When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages","created":"2023-11-15","cited_by_count":32},{"id":"2405.10853","title":"The Future of Large Language Model Pre-training is Federated","created":"2024-05-17","cited_by_count":30},{"id":"2307.09619","title":"Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning","created":"2023-07-18","cited_by_count":29},{"id":"2112.03497","title":"Dataset Geography: Mapping Language Data to Language Users","created":"2021-12-07","cited_by_count":22},{"id":"2110.08133","title":"Trade-offs of Local SGD at Scale: An Empirical Study","created":"2021-10-15","cited_by_count":20},{"id":"2305.15020","title":"An Efficient Multilingual Language Model Compression through Vocabulary Trimming","created":"2023-05-24","cited_by_count":20},{"id":"2112.14569","title":"Fine-Tuning Transformers: Vocabulary Transfer","created":"2021-12-29","cited_by_count":14},{"id":"2305.13286","title":"How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning","created":"2023-05-22","cited_by_count":14},{"id":"2402.05968","title":"Federated Learning Priorities Under the European Union Artificial Intelligence Act","created":"2024-02-05","cited_by_count":13}],"next_cursor":null}