mlsanityData to 7 Oct 2026

Exploring the Limits of Language Modeling

arXiv 1602.02410 ↗First posted 7 Feb 2016cs.CLcited by 172

Not on the map. Why?

Abstract (arXiv)

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 0; 2020: 0; 2021: 50; 2022: 27; 2023: 35; 2024: 35; 2025: 15; 2026: 8.

172 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
38 more
  1. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  2. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  4. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  5. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  6. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  7. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  9. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  10. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  11. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  12. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  13. 2022Video Transformers: A Surveycited by 63
  14. 2021Video Corpus Moment Retrieval with Contrastive Learningcited by 60
  15. 2021A Survey on Green Deep Learningcited by 52
  16. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  17. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  18. 2021Multi-view Subword Regularizationcited by 40
  19. 2023Contrastive Difference Predictive Codingcited by 35
  20. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  21. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  22. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  23. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  24. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  25. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  26. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Modelscited by 29
  27. 2022From t-SNE to UMAP with contrastive learningcited by 28
  28. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  29. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  30. 2022Interpolation-based Contrastive Learning for Few-Label Semi-Supervised Learningcited by 23
  31. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  32. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  33. 2024Differentially Private Next-Token Prediction of Large Language Modelscited by 21
  34. 2021Joint Multimedia Event Extraction from Video and Articlecited by 19
  35. 2023Accommodating Audio Modality in CLIP for Multimodal Processingcited by 18
  36. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  37. 2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
  38. 2021A Survey on Low-Resource Neural Machine Translationcited by 16

50 most cited of 172 All 172 (JSON)