Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
In today's artificial intelligence driven world, modern systems communicate with people from diverse backgrounds and skill levels. For human-machine interaction to be meaningful, systems must be aware of context and user expertise. This study proposes an agentic AI profiler that classifies natural language responses into four levels: Novice, Basic, Advanced, and Expert. The system uses a modular layered architecture built on LLaMA v3.1 (8B), with components for text preprocessing, scoring, aggregation, and classification. Evaluation was conducted in two phases: a static phase using pre-recorded transcripts from 82 participants, and a dynamic phase with 402 live interviews conducted by an agentic AI interviewer. In both phases, participant self-ratings were compared with profiler predictions. In the dynamic phase, expertise was assessed after each response rather than at the end of the interview. Across domains, 83% to 97% of profiler evaluations matched participant self-assessments. Remaining differences were due to self-rating bias, unclear responses, and occasional misinterpretation of nuanced expertise by the language model.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 6 of its 18 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
6 of 18 references.
References · 6
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
- 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
- 2024Personalized Large Language Modelscited by 44
- 2022Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Makingcited by 33
- 2025Tailored Conversations beyond LLMs: A RL-Based Dialogue Managercited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- 2025Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context
- 2026Overview of the TREC 2025 Million Large Language Models track
- 2024Characteristic AI Agents via Large Language Models
- 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
- 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
- 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
- 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competencies