mlsanityData to 7 Oct 2026

Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning

arXiv 2406.10908 ↗First posted 16 Jun 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

22 of 40 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  10. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  11. 2022Active Example Selection for In-Context Learningcited by 195
  12. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
10 more
  1. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  2. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  3. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  4. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  5. 2023Finding Support Examples for In-Context Learningcited by 104
  6. 2023Mitigating Label Biases for In-context Learningcited by 88
  7. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  8. 2023In-Context Learning for Text Classification with Many Labelscited by 35
  9. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18
  10. 2024Naive Bayes-based Context Extension for Large Language Modelscited by 5

Built on by

Not cited yet.