mlsanityData to 14 Oct 2026

LinguistAgent Technical Report: A Reflective Multi-Model Platform for Automated Linguistic Annotation

arXiv 2602.05493 ↗First posted 5 Feb 2026cs.CL cs.AI cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Data annotation remains a significant bottleneck in the field of humanities and social sciences, particularly for complex linguistic tasks such as metaphor identification. While Large Language Models (LLMs) show promise, a significant gap remains between the theoretical capability of LLMs and their practical utility for researchers. This paper introduces LinguistAgent, an integrated, user-friendly platform that leverages a reflective multi-model architecture to automate linguistic annotation. The platform comprises an Annotator and an optional Reviewer to simulate a peer-review process. This platform supports comparative experiments across three main paradigms: Prompt Engineering (Zero-shot/Few-shot/Chain-of-thought), Retrieval-Augmented Generation, and Fine-tuning. We demonstrate LinguistAgent's efficacy by replicating the task of metaphor identification from a published study, which provides real-time token-level evaluation (F1 and Cohen's kappa) against human gold standards. The application and codes are released on https://github.com/Bingru-Li/LinguistAgent.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 2 of its 17 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

2 of 17 references.

References · 2

  1. 2024Science is Exploration: Computational Frontiers for Conceptual Metaphor Theorycited by 4
  2. 2025Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation System
  2. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
  3. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
  4. 2026Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery
  5. 2025Evaluating Large Language Models as Expert Annotators
  6. 2024MEGAnno+: A Human-LLM Collaborative Annotation System
  7. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  8. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery