mlsanityData to 7 Oct 2026

Evaluation of medium-large Language Models at zero-shot closed book generative question answering

arXiv 2305.11991 ↗First posted 19 May 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have garnered significant attention, but the definition of "large" lacks clarity. This paper focuses on medium-sized language models (MLMs), defined as having at least six billion parameters but less than 100 billion. The study evaluates MLMs regarding zero-shot generative question answering, which requires models to provide elaborate answers without external document retrieval. The paper introduces an own test dataset and presents results from human evaluation. Results show that combining the best answers from different MLMs yielded an overall correct answer rate of 82.7% which is better than the 60.9% of ChatGPT. The best MLM achieved 71.8% and has 33B parameters, which highlights the importance of using appropriate training data for fine-tuning rather than solely relying on the number of parameters. More fine-grained feedback should be used to further improve the quality of answers. The open source community is quickly closing the gap to the best commercial models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

42 of 63 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
30 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  14. 2023Instruction Tuning with GPT-4cited by 676
  15. 2022Large Language Models Can Self-Improvecited by 656
  16. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  17. 2019ELI5: Long Form Question Answeringcited by 567
  18. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  19. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  20. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  21. 2022Teaching Small Language Models to Reasoncited by 279
  22. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  23. 2022Language models show human-like content effects on reasoning taskscited by 171
  24. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  25. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  26. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  27. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  28. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  29. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  30. 2022Query Refinement Prompts for Closed-Book Long-Form Question Answeringcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  2. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49