# Verb-ICL: Rethinking In-Context Learning for Structured Prediction

arXiv: https://arxiv.org/abs/2610.04725 · HTML: https://mlsanity.com/paper/2610.04725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 38 of its 72 references that are in the map.
- First posted on arXiv 3 Oct 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 38)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [In-context Pretraining: Language Modeling Beyond Document Boundaries](https://mlsanity.com/paper/2310.10638/) (2023)
- [Coverage-based Example Selection for In-Context Learning](https://mlsanity.com/paper/2305.14907/) (2023)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [In-Context Principle Learning from Mistakes](https://mlsanity.com/paper/2402.05403/) (2024)
- [Pre-Training to Learn in Context](https://mlsanity.com/paper/2305.09137/) (2023)
- [Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection](https://mlsanity.com/paper/2310.20046/) (2023)
- [Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts](https://mlsanity.com/paper/2210.03797/) (2022)
- [IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models](https://mlsanity.com/paper/2310.10873/) (2023)
- [On Training Instance Selection for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2107.03176/) (2021)
- [Schema-Driven Information Extraction from Heterogeneous Tables](https://mlsanity.com/paper/2305.14336/) (2023)
- [Active Learning for Interactive Neural Machine Translation of Data Streams](https://mlsanity.com/paper/1807.11243/) (2018)
- [From Selection to Generation: A Survey of LLM-based Active Learning](https://mlsanity.com/paper/2502.11767/) (2025)
- [A Survey to Recent Progress Towards Understanding In-Context Learning](https://mlsanity.com/paper/2402.02212/) (2024)
- [ELAD: Explanation-Guided Large Language Models Active Distillation](https://mlsanity.com/paper/2402.13098/) (2024)
- [Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study](https://mlsanity.com/paper/2402.15663/) (2024)
- [To Label or Not to Label: Hybrid Active Learning for Neural Machine Translation](https://mlsanity.com/paper/2403.09259/) (2024)

## Abstract (arXiv)

Structured prediction tasks pose unique challenges for in-context learning (ICL): their compositional outputs require modeling fine-grained, token-level patterns that sentence-level approaches fail to capture, and their task-specific annotation conventions are human-defined artifacts that cannot be acquired through pretraining alone. We propose Verb-ICL, a selective annotation framework for ICL-based structured prediction that addresses both challenges. Verb-ICL first selects representative examples using a token-level coverage strategy that captures local semantic patterns critical for structured prediction, then generates actionable error feedback that codifies task-specific annotation guidelines and incorporates this feedback into ICL demonstrations. We evaluate Verb-ICL on six structured prediction datasets spanning information extraction and semantic parsing. Experiments with recent LLMs show that Verb-ICL consistently outperforms strong selective annotation baselines under low-resource settings and continues to provide gains as the annotation budget increases. Extended analyses demonstrate that the generated feedback is predominantly useful across a four-category quality taxonomy, generalizes as task-level guidance beyond instance-specific corrections, and improves performance regardless of the underlying selection strategy.