mlsanityData to 7 Oct 2026

On the State of the Art of Evaluation in Neural Language Models

arXiv 1707.05589 ↗First posted 18 Jul 2017cs.CLcited by 103

Not on the map. Why?

Abstract (arXiv)

Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 35; 2022: 16; 2023: 22; 2024: 15; 2025: 4; 2026: 10.

103 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  6. 2021Autoregressive Diffusion Modelscited by 208
  7. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  8. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  9. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  10. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  11. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  12. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
38 more
  1. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62
  2. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  3. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  4. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  5. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  6. 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 34
  7. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  8. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  9. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  10. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  11. 2021Selfish Sparse RNN Trainingcited by 26
  12. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  13. 2021Automatic Termination for Hyperparameter Optimizationcited by 21
  14. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19
  15. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  16. 2022Measuring Datacited by 18
  17. 2023Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directionscited by 18
  18. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  19. 2024G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detectioncited by 17
  20. 2023PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspacescited by 15
  21. 2021Reproducibility in Evolutionary Computationcited by 12
  22. 2021Context-Aware Legal Citation Recommendation using Deep Learningcited by 12
  23. 2021A Cognitive Regularizer for Language Modelingcited by 11
  24. 2022Modeling the Machine Learning Multiversecited by 11
  25. 2023Lo-Hi: Practical ML Drug Discovery Benchmarkcited by 10
  26. 2024Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRLcited by 10
  27. 2021Is it enough to optimize CNN architectures on ImageNet?cited by 9
  28. 2021Perspectives on Machine Learning from Psychology's Reproducibility Crisiscited by 9
  29. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9
  30. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  31. 2023DA-LSTM: A Dynamic Drift-Adaptive Learning Framework for Interval Load Forecasting with LSTM Networkscited by 8
  32. 2021Modeling the Unigram Distributioncited by 7
  33. 2021Language Models as a Knowledge Source for Cognitive Agentscited by 7
  34. 2023On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Researchcited by 7
  35. 2023SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecastingcited by 7
  36. 2021Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformerscited by 6
  37. 2021Expected Validation Performance and Estimation of a Random Variable's Maximumcited by 6
  38. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 6

50 most cited of 103 All 103 (JSON)