mlsanityData to 7 Oct 2026

Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities

arXiv 2402.10424 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a simple theoretical framework, Pelican Soup, aiming to better understand how pretraining allows LLMs to (1) generalize to unseen instructions and (2) perform in-context learning, even when the verbalizers are irrelevant to the task. To this end, in our framework, we introduce the notion of "knowledge base" and "reference-sense association" and a simple formalism for natural language processing tasks. Our framework demonstrates how linguistic, psychology, and philosophy studies can inform our understanding of the language model and is connected to several other existing theoretical results. As an illustration of the usage of our framework, we derive a bound on in-context learning loss with our framework. Finally, we support our framework with empirical experiments and provide possible future research directions.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 64 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

35 of 64 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2023Larger language models do in-context learning differentlycited by 386
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
23 more
  1. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  2. 2023In-Context Learning Creates Task Vectorscited by 302
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  4. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  5. 2022Can language models learn from explanations in context?cited by 248
  6. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  7. 2021Meta-learning via Language Model In-context Tuningcited by 163
  8. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  9. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  10. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  11. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  12. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  13. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  14. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  15. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  16. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  17. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  18. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  19. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  20. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  21. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  22. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 56
  23. 2023On Retrieval Augmentation and the Limitations of Language Model Trainingcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoningcited by 27
  2. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3