# STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking

arXiv: https://arxiv.org/abs/2507.03674 · HTML: https://mlsanity.com/paper/2507.03674/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 4 Jul 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [BM25S: Orders of magnitude faster lexical search via eager sparse scoring](https://mlsanity.com/paper/2407.03618/) (2024)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [TRIE: End-to-End Text Reading and Information Extraction for Document Understanding](https://mlsanity.com/paper/2005.13118/) (2020)
- [RAH! RecSys-Assistant-Human: A Human-Centered Recommendation Framework with LLM Agents](https://mlsanity.com/paper/2308.09904/) (2023)
- [Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts](https://mlsanity.com/paper/1808.06161/) (2018)
- [InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration](https://mlsanity.com/paper/2402.11441/) (2024)
- [Efficient Knowledge Infusion via KG-LLM Alignment](https://mlsanity.com/paper/2406.03746/) (2024)
- [Scientific Discourse Tagging for Evidence Extraction](https://mlsanity.com/paper/1909.04758/) (2019)
- [Do LLMs Really Adapt to Domains? An Ontology Learning Perspective](https://mlsanity.com/paper/2407.19998/) (2024)
- [NCBO Ontology Recommender 2.0: An Enhanced Approach for Biomedical Ontology Recommendation](https://mlsanity.com/paper/1611.05973/) (2016)
- [LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop](https://mlsanity.com/paper/2402.09346/) (2024)
- [Experiment Segmentation in Scientific Discourse as Clause-level Structured Prediction using Recurrent Neural Networks](https://mlsanity.com/paper/1702.05398/) (2017)

## Built on by (most cited first, 8 of 8)

- [AI Can Learn Scientific Taste](https://mlsanity.com/paper/2603.14473/) (2026)
- [Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents](https://mlsanity.com/paper/2511.08242/) (2025)
- [A System for Name and Address Parsing with Large Language Models](https://mlsanity.com/paper/2601.18014/) (2026)
- [From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFs](https://mlsanity.com/paper/2601.14267/) (2025)
- [Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence](https://mlsanity.com/paper/2608.25869/) (2026)
- [When Evidence Shapes Collaboration: Knowledge-Conditioned Topology Generation for Multi-Agent Systems](https://mlsanity.com/paper/2608.27984/) (2026)
- [FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases](https://mlsanity.com/paper/2602.09163/) (2026)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)

## Abstract (arXiv)

Extracting structured information from scientific literature is critical for accelerating discovery, yet Large Language Models (LLMs) often struggle in specialized domains that require expert knowledge and generalize poorly across tasks. We introduce textscStructSense, a modular, task-agnostic, open-source framework that integrates ontology-guided symbolic knowledge, agentic self-evaluative refinement, and human-in-the-loop validation for robust domain-aware extraction. We evaluate textscStructSense on three tasks of increasing semantic complexity: schema-based extraction of assessment instruments (91--100% accuracy), metadata and resource extraction from scientific papers (86--93% overall), and named entity recognition (NER) from neuroscience literature (58--75% label accuracy across 8,882 entities). On two biomedical NER benchmarks (NCBI Disease and S800 Species), the system achieves ≥90% relaxed recall and 62.5--85.8% strict recall while extracting 1,000--3,600 additional entities beyond gold annotations. The local concept mapping service achieves Hits@1 of 62--82% under strict matching and 68--86% under semantic matching. These results across three domains demonstrate that textscStructSense generalizes across tasks while maintaining source grounding and provenance transparency.