mlsanityData to 7 Oct 2026

LMentry: A Language Model Benchmark of Elementary Language Tasks

arXiv 2211.02069 ↗First posted 3 Nov 2022cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

As the performance of large language models rapidly improves, benchmarks are getting larger and more complex as well. We present LMentry, a benchmark that avoids this "arms race" by focusing on a compact set of tasks that are trivial to humans, e.g. writing a sentence containing a specific word, identifying which words in a list belong to a specific category, or choosing which of two words is longer. LMentry is specifically designed to provide quick and interpretable insights into the capabilities and robustness of large language models. Our experiments reveal a wide variety of failure cases that, while immediately obvious to humans, pose a considerable challenge for large language models, including OpenAI's latest 175B-parameter instruction-tuned model, TextDavinci002. LMentry complements contemporary evaluation approaches of large language models, providing a quick, automatic, and easy-to-run "unit test", without resorting to large benchmark suites of complex tasks.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

32 of 45 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
20 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  9. 2021True Few-Shot Learning with Language Modelscited by 343
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  11. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  12. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  13. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  14. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  15. 2022Predictability and Surprise in Large Generative Modelscited by 209
  16. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  17. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  18. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  19. 2021Language Model Evaluation Beyond Perplexitycited by 77
  20. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 4; 2025: 6; 2026: 3.

21 papers.

Sort:

By area:

  1. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  2. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  3. 2023How is ChatGPT's behavior changing over time?cited by 265
  4. 2024Efficient multi-prompt evaluation of LLMscited by 72
  5. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  6. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  7. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 21
  8. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  9. 2023Surfacing Biases in Large Language Models using Contrastive Input Decodingcited by 11
  10. 2023Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Featurescited by 6
  11. 2024Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Frameworkcited by 4
  12. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologiescited by 3
9 more
  1. 2025CharBench: Evaluating the Role of Tokenization in Character-Level Taskscited by 3
  2. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Taskscited by 3
  3. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
  4. 2025Re-Evaluating Code LLM Benchmarks Under Semantic Mutationcited by 1
  5. 2025Position: AI Evaluations Should be Grounded on a Theory of Capabilitycited by 1
  6. 2023Transcending the Attention Paradigm: Representation Learning from Geospatial Social Media Datacited by 0
  7. 2025TASE: Token Awareness and Structured Evaluation for Multilingual Language Modelscited by 0
  8. 2026SubTokenTest: A Practical Benchmark for Real-World Sub-token Understandingcited by 0
  9. 2026Headlines You Won't Forget: Can Pronoun Insertion Increase Memorability?cited by 0