mlsanityData to 14 Oct 2026

Medal Matters: Probing LLMs' Failure Cases Through Olympic Rankings

arXiv 2409.06518 ↗First posted 10 Sep 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable success in natural language processing tasks, yet their internal knowledge structures remain poorly understood. This study examines these structures through the lens of historical Olympic medal tallies, evaluating LLMs on two tasks: (1) retrieving medal counts for specific teams and (2) identifying rankings of each team. While state-of-the-art LLMs excel in recalling medal counts, they struggle with providing rankings, highlighting a key difference between their knowledge organization and human reasoning. These findings shed light on the limitations of LLMs' internal knowledge integration and suggest directions for improvement. To facilitate further research, we release our code, dataset, and model outputs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

19 of 30 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  11. 2024Large Language Models: A Surveycited by 525
  12. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
7 more
  1. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  2. 2024The pitfalls of next-token predictioncited by 160
  3. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  4. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  5. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  6. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  7. 2024On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoningcited by 26

Built on by

Not cited yet.