# Can AI Guess What You Know? Performance Comparison of Large Language Models for Human Domain Knowledge Estimation From Communication Logs

arXiv: https://arxiv.org/abs/2605.22971 · HTML: https://mlsanity.com/paper/2605.22971/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 12 of 12)

- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs](https://mlsanity.com/paper/2307.10168/) (2023)
- [The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends](https://mlsanity.com/paper/2409.14195/) (2024)
- [LLMs-Healthcare : Current Applications and Challenges of Large Language Models in various Medical Specialties](https://mlsanity.com/paper/2311.12882/) (2023)
- [GazeLLM: Multimodal LLMs incorporating Human Visual Attention](https://mlsanity.com/paper/2504.00221/) (2025)
- [GenAIReading: Augmenting Human Cognition with Interactive Digital Textbooks Using Large Language Models and Image Generation Models](https://mlsanity.com/paper/2503.07463/) (2025)
- [Exploring the Design Space of Real-time LLM Knowledge Support Systems: A Case Study of Jargon Explanations](https://mlsanity.com/paper/2503.00715/) (2025)
- [Augmented Body Communicator: Enhancing daily body expression for people with upper limb limitations through LLM and a robotic arm](https://mlsanity.com/paper/2505.05832/) (2025)
- [Towards Human-AI Mutual Learning: A New Research Paradigm](https://mlsanity.com/paper/2405.04687/) (2024)
- [Leveraging Knowledge Graphs and LLMs for Context-Aware Messaging](https://mlsanity.com/paper/2503.13499/) (2025)
- [Augmenting Online Meetings with Context-Aware Real-time Music Generation](https://mlsanity.com/paper/2503.01354/) (2025)

## Abstract (arXiv)

Employees often struggle to identify ``who knows what,'' leading to organizational productivity losses. We investigate whether Large Language Models (LLMs) can infer individual domain knowledge directly from long-term Slack logs. Analyzing 27,188 messages from 43 users, we evaluated seven models (including Gemini, Claude, and GPT families) by comparing their zero-shot estimates against self-reported skill ratings from 27 participants. Gemini 2.5 Flash achieved the lowest error (MAE 21.13%), while GPT models showed significantly larger discrepancies. Notably, estimation accuracy depended only weakly on message volume, indicating that more text alone does not guarantee better inference. These findings demonstrate the feasibility and current limits of automated expertise mapping, highlighting the need for privacy-preserving deployments and richer, structure-aware representations of human knowledge.