mlsanityData to 14 Oct 2026

An Annotation Scheme and Classifier for Personal Facts in Dialogue

arXiv 2605.10339 ↗First posted 11 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The advancement of Large Language Models (LLMs) has enabled their application in personalized dialogue systems. We present an extended annotation scheme for personal fact classification that addresses limitations in existing approaches, particularly PeaCoK. Our scheme introduces new categories (Demographics, Possessions) and attributes (Duration, Validity, Followup) that enable structured storage, quality filtering, and identification of facts suitable for dialogue continuation. We manually annotated 2,779 facts from Multi-Session Chat and trained a multi-head classifier based on transformer encoders. Combined with the Gemma-300M encoder, the classifier achieves 81.6 ± 2.6% macro F1, outperforming all few-shot LLM baselines (best: GPT-5.4-mini, 72.92%) by nearly 9 percentage points while requiring substantially fewer computational resources. Error analysis reveals persistent challenges in semantic boundary disambiguation, temporal aspect interpretation, and pragmatic reasoning for followup assessment. The datasetfootnotemark[1] and classifierfootnotemark[2] are publicly available.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

39 of 60 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  10. 2025OpenAI GPT-5 System Cardcited by 989
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
27 more
  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  3. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  4. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  5. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  6. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  8. 2022Is GPT-3 a Good Data Annotator?cited by 208
  9. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  10. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  11. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  12. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  13. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  14. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  15. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 123
  16. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  17. 2019Large-Scale Multi-Label Text Classification on EU Legislationcited by 104
  18. 2018AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classificationcited by 102
  19. 2018SGM: Sequence Generation Model for Multi-label Classificationcited by 89
  20. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  21. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 89
  22. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 71
  23. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  24. 2023Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Personacited by 24
  25. 2023PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narrativescited by 19
  26. 2024Learning Retrieval Augmentation for Personalized Dialogue Generationcited by 18
  27. 2024Selective Prompting Tuning for Personalized Conversations with LLMscited by 16

Built on by

Not cited yet.