mlsanityData to 14 Oct 2026

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions

arXiv 2502.13791 ↗First posted 19 Feb 2025cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

26 of 48 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  10. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  11. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
14 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  3. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  4. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  5. 2023Harnessing large-language models to generate private synthetic textcited by 65
  6. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  7. 2023Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversationscited by 47
  8. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  9. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 34
  10. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMscited by 20
  11. 2022MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussionscited by 17
  12. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  13. 2024SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Dataset Constructed from Movie Scriptcited by 7
  14. 2024Crystal: Illuminating LLM Abilities on Language and Codecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  2. 2026Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agentscited by 7
  3. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6
  4. 2026SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Historiescited by 1
  5. 2026UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilitiescited by 0
  6. 2026DolphinBench: Mapping the Pareto Frontier of Agent Memorycited by 0
  7. 2026Learning from Others, Acting for You: Cross-User Memory Sharing for LLM Agentscited by 0
  8. 2026Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agentscited by 0