# LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History

arXiv: https://arxiv.org/abs/2402.18216 · HTML: https://mlsanity.com/paper/2402.18216/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [An LLM can Fool Itself: A Prompt-Based Adversarial Attack](https://mlsanity.com/paper/2310.13345/) (2023)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [A Theory of Emergent In-Context Learning as Implicit Structure Induction](https://mlsanity.com/paper/2303.07971/) (2023)
- [Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning](https://mlsanity.com/paper/2305.17256/) (2023)
- [Data Curation Alone Can Stabilize In-context Learning](https://mlsanity.com/paper/2212.10378/) (2022)
- [TWEETQA: A Social Media Focused Question Answering Dataset](https://mlsanity.com/paper/1907.06292/) (2019)
- [Transformers generalize differently from information stored in context vs in weights](https://mlsanity.com/paper/2210.05675/) (2022)
- [Topic-Aware Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2009.12539/) (2020)
- [Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks](https://mlsanity.com/paper/2310.10077/) (2023)
- [SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems](https://mlsanity.com/paper/2110.06800/) (2021)

## Built on by (most cited first, 19 of 19)

- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directions](https://mlsanity.com/paper/2407.04581/) (2024)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction](https://mlsanity.com/paper/2603.04783/) (2026)
- [OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models](https://mlsanity.com/paper/2508.21061/) (2025)
- [Mitigating Conversational Inertia in Multi-Turn Agents](https://mlsanity.com/paper/2602.03664/) (2026)
- [ERGO: Entropy-guided Resetting for Generation Optimization in Multi-turn Language Models](https://mlsanity.com/paper/2510.14077/) (2025)
- [Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History](https://mlsanity.com/paper/2505.21362/) (2025)
- [Cognitive Load Limits in Large Language Models: Benchmarking Multi-Hop Reasoning](https://mlsanity.com/paper/2509.19517/) (2025)
- [A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities](https://mlsanity.com/paper/2603.02540/) (2026)
- [Vision-Speech Models: Teaching Speech Models to Converse about Images](https://mlsanity.com/paper/2503.15633/) (2025)
- [Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents](https://mlsanity.com/paper/2510.14453/) (2025)
- [Persuasion Propagation: Does Persuasion Change AI Agent Behavior?](https://mlsanity.com/paper/2602.00851/) (2026)
- [APWA: A Distributed Architecture for Parallelizable Agentic Workflows](https://mlsanity.com/paper/2605.15132/) (2026)
- [SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories](https://mlsanity.com/paper/2608.09799/) (2026)
- [When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents](https://mlsanity.com/paper/2510.22963/) (2025)
- [Retcon -- a Prompt-Based Technique for Precise Control of LLMs in Conversations](https://mlsanity.com/paper/2603.03317/) (2026)
- [Multimodal Task Interference: A Benchmark and Analysis of History-Target Mismatch in Multimodal LLMs](https://mlsanity.com/paper/2603.18425/) (2026)
- [At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization](https://mlsanity.com/paper/2606.26396/) (2026)

## Abstract (arXiv)

With the recent emergence of powerful instruction-tuned large language models (LLMs), various helpful conversational Artificial Intelligence (AI) systems have been deployed across many applications. When prompted by users, these AI systems successfully perform a wide range of tasks as part of a conversation. To provide some sort of memory and context, such approaches typically condition their output on the entire conversational history. Although this sensitivity to the conversational history can often lead to improved performance on subsequent tasks, we find that performance can in fact also be negatively impacted, if there is a task-switch. To the best of our knowledge, our work makes the first attempt to formalize the study of such vulnerabilities and interference of tasks in conversational LLMs caused by task-switches in the conversational history. Our experiments across 5 datasets with 15 task switches using popular LLMs reveal that many of the task-switches can lead to significant performance degradation.