mlsanityData to 7 Oct 2026

The LAMBADA dataset: Word prediction requiring a broad discourse context

arXiv 1606.06031 ↗First posted 20 Jun 2016cs.CL cs.AI cs.LGcited by 960

Not on the map. Why?

Cited by 66 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

Key facts. Cited by 960 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 49; 2023: 99; 2024: 227; 2025: 262; 2026: 291.

960 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  15. 2023A Comprehensive Overview of Large Language Modelscited by 639
  16. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  17. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  18. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  19. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  20. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  21. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  22. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  23. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  24. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  25. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  26. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  27. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  28. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  29. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  30. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  31. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  32. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  33. 2024Titans: Learning to Memorize at Test Timecited by 303
  34. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  35. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  36. 2023A Survey on Model Compression for Large Language Modelscited by 263
  37. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  38. 2022Unified Scaling Laws for Routed Language Modelscited by 249

50 most cited of 960 All 960 (JSON)