mlsanityData to 7 Oct 2026

OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

arXiv 2501.15030 ↗First posted 25 Jan 2025cs.LG cs.AI cs.CLcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 37 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

26 of 37 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2021True Few-Shot Learning with Language Modelscited by 343
  11. 2024Many-Shot In-Context Learningcited by 203
  12. 2021Meta-learning via Language Model In-context Tuningcited by 163
14 more
  1. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  2. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  3. 2023Compositional Exemplars for In-context Learningcited by 143
  4. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  5. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  6. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  7. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  8. 2023Prompt Optimization via Adversarial In-Context Learningcited by 33
  9. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  10. 2024Is In-Context Learning Sufficient for Instruction Following in LLMs?cited by 19
  11. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  12. 2024Se2: Sequential Example Selection for In-Context Learningcited by 8
  13. 2024EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoningcited by 5
  14. 2024One size doesn't fit all: Predicting the Number of Examples for In-Context Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  2. 2025The Impossibility of Inverse Permutation Learning in Transformer Modelscited by 2
  3. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
  4. 2026Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamicscited by 1
  5. 2025On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detectioncited by 0
  6. 2026Self-Improving In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Order Matters: Rethinking Prompt Construction in In-Context Learning
  2. 2024Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement
  3. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs
  4. 2023In-Context Explainers: Harnessing LLMs for Explaining Black Box Models
  5. 2024Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMs
  6. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
  7. 2025Leveraging In-Context Learning for Language Model Agents
  8. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning