mlsanityData to 14 Oct 2026

TPBench: A Turning-Point Benchmark for Dialogue Compression

arXiv 2610.02736 ↗First posted 2 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A compressor can keep the facts of a dialogue and still drop the turn that changed them. A user corrects a price, reverses a choice, or adds a constraint. We call this failure turning-point eviction. One overall retention score hides it, because that score mixes what the user first wanted with what the user wants now. We introduce TPBench, which evaluates three complementary information targets at shared nominal retention budgets. P1 asks for the user's initial goal. P2 asks for the current value of a slot the user revised. P3 asks for both, in dialogues with a late annotated slot update. The current-value answers come from the human dialogue-state annotations of MultiWOZ and SGD. The initial-goal answer is the first sentence of the first user turn. Neither requires new crowdsourcing. The probe-specific evaluations rank compression methods differently. On the joint probe at a retained fraction of 0.30, every tested compressed method remains below full context with the main Llama reader. Deleting the turn that carries the update sharply lowers current-value accuracy, while deleting one matched irrelevant turn leaves it unchanged. A Mistral reader repeats the P2/P3 rankings and the joint-probe gap. Current-value recovery is tested on an additional corpus, LongMemEval-KU, and on Chinese RiSAWOZ: full context has the highest accuracy, and recency has the highest compressed-method mean in both evaluations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  12. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
27 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  4. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  7. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  9. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 216
  10. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 204
  11. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  12. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 131
  13. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  14. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  15. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  16. 2024Prompt Compression for Large Language Models: A Surveycited by 67
  17. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  18. 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 54
  19. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  20. 2020RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modelingcited by 41
  21. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  22. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 19
  23. 2024CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Modelscited by 15
  24. 2022Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performancescited by 14
  25. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 14
  26. 2024Mixed-Session Conversation with Egocentric Memorycited by 8
  27. 2023Unsupervised Dialogue Topic Segmentation in Hyperdimensional Spacecited by 4

Built on by

Not cited yet.