# Serial Position Effects of Large Language Models

arXiv: https://arxiv.org/abs/2406.15981 · HTML: https://mlsanity.com/paper/2406.15981/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Black Box Adversarial Prompting for Foundation Models](https://mlsanity.com/paper/2302.04237/) (2023)
- [Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach](https://mlsanity.com/paper/2310.08172/) (2023)

## Built on by (most cited first, 30 of 30)

- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs](https://mlsanity.com/paper/2510.10276/) (2025)
- [Playing games with Large language models: Randomness and strategy](https://mlsanity.com/paper/2503.02582/) (2025)
- [Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models](https://mlsanity.com/paper/2506.07334/) (2025)
- [CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants](https://mlsanity.com/paper/2410.21159/) (2024)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [A Structural Theory of Position Bias in Transformers](https://mlsanity.com/paper/2602.16837/) (2026)
- [Emergence of the Primacy Effect in Structured State-Space Models](https://mlsanity.com/paper/2502.13729/) (2025)
- [Attention Debiasing for Token Pruning in Vision Language Models](https://mlsanity.com/paper/2508.17807/) (2025)
- [Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG](https://mlsanity.com/paper/2506.04390/) (2025)
- [Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents](https://mlsanity.com/paper/2510.14453/) (2025)
- [Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments](https://mlsanity.com/paper/2512.19154/) (2025)
- [AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications](https://mlsanity.com/paper/2512.20164/) (2025)
- [On the Reasoning Capacity of AI Models and How to Quantify It](https://mlsanity.com/paper/2501.13833/) (2025)
- [Analysis of LLM as a grammatical feature tagger for African American English](https://mlsanity.com/paper/2502.06004/) (2025)
- [How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization](https://mlsanity.com/paper/2607.07678/) (2026)
- [On Psychology of AI -- Does Primacy Effect Affect ChatGPT and Other LLMs?](https://mlsanity.com/paper/2504.20444/) (2025)
- [To Copy or Not to Copy: Copying Is Easier to Induce Than Recall](https://mlsanity.com/paper/2601.12075/) (2026)
- [PLR: Plackett-Luce for Reordering In-Context Learning Examples](https://mlsanity.com/paper/2603.21373/) (2026)
- [Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation](https://mlsanity.com/paper/2604.19777/) (2026)
- [Token Homogenization under Positional Bias](https://mlsanity.com/paper/2508.17126/) (2025)
- [Approximating Human Preferences Using a Multi-Judge Learned System](https://mlsanity.com/paper/2510.25884/) (2025)
- [Interpretable-by-Design Transformers via Architectural Stream Independence](https://mlsanity.com/paper/2603.07482/) (2026)
- [SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimization](https://mlsanity.com/paper/2604.17351/) (2026)
- [@skills: Attention is all you have](https://mlsanity.com/paper/2608.12610/) (2026)
- [The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systems](https://mlsanity.com/paper/2609.03425/) (2026)
- [SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs](https://mlsanity.com/paper/2609.10901/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities in zero-shot learning applications, generating responses to queries using only pre-training information without the need for additional fine-tuning. This represents a significant departure from traditional machine learning approaches. Previous research has indicated that LLMs may exhibit serial position effects, such as primacy and recency biases, which are well-documented cognitive biases in human psychology. Our extensive testing across various tasks and models confirms the widespread occurrence of these effects, although their intensity varies. We also discovered that while carefully designed prompts can somewhat mitigate these biases, their effectiveness is inconsistent. These findings underscore the significance of serial position effects during the inference process, particularly in scenarios where there are no ground truth labels, highlighting the need for greater focus on addressing these effects in LLM applications.