mlsanityData to 7 Oct 2026

Order Matters: Rethinking Prompt Construction in In-Context Learning

arXiv 2511.09700 ↗First posted 12 Nov 2025cs.CLcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples. Most prior work reasonably and intuitively assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered, leading to a focus on example selection. We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering. Through controlled experiments on both classification and generation tasks, using multiple open-source model families (0.5B to 27B parameters) and GPT-5, we find that the variance in performance due to different example orderings is comparable to that from using entirely different example sets. Furthermore, we show that strong orderings can be identified using only a development set, achieving performance close to an oracle that selects the best ordering based on test labels. Our findings highlight the equal and intertwined importance of example selection and ordering in prompt design, calling for a reexamination of the assumptions held in ICL.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 23 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

9 of 23 references.

References · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  9. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamicscited by 1
  2. 2026When Context Sticks: Studying Interference in In-Context Learningcited by 0
  3. 2026Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learning
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examples
  4. 2024Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions
  5. 2026Not the Example, but the Process: How Self-Generated Examples Enhance LLM Reasoning
  6. 2025The Impact of Role Design in In-Context Learning for Large Language Models
  7. 2025Is In-Context Learning Learning?
  8. 2025On the Relationship Between the Choice of Representation and In-Context Learning