mlsanityData to 13 Oct 2026

TEL'M: Test and Evaluation of Language Models

arXiv 2404.10200 ↗First posted 16 Apr 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Language Models have demonstrated remarkable capabilities on some tasks while failing dramatically on others. The situation has generated considerable interest in understanding and comparing the capabilities of various Language Models (LMs) but those efforts have been largely ad hoc with results that are often little more than anecdotal. This is in stark contrast with testing and evaluation processes used in healthcare, radar signal processing, and other defense areas. In this paper, we describe Test and Evaluation of Language Models (TEL'M) as a principled approach for assessing the value of current and future LMs focused on high-value commercial, government and national security applications. We believe that this methodology could be applied to other Artificial Intelligence (AI) technologies as part of the larger goal of "industrializing" AI.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

35 of 71 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  8. 2023A Survey of Large Language Modelscited by 2,559
  9. 2022Emergent Abilities of Large Language Modelscited by 2,499
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,286
23 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,236
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 652
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 602
  4. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 416
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 334
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 313
  7. 2018Metrics for Explainable AI: Challenges and Prospectscited by 258
  8. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 180
  9. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 154
  10. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  11. 2023What Formal Languages Can Transformers Express? A Surveycited by 121
  12. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 111
  13. 2017A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortionscited by 107
  14. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  15. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 69
  16. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  17. 2023The Foundation Model Transparency Indexcited by 64
  18. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  19. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  20. 2023On the Creativity of Large Language Modelscited by 57
  21. 2022Transformers in Time-series Analysis: A Tutorialcited by 46
  22. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  23. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyondcited by 14

Built on by

Not cited yet.