# Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution

arXiv: https://arxiv.org/abs/2503.03201 · HTML: https://mlsanity.com/paper/2503.03201/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [Data Augmentation Can Improve Robustness](https://mlsanity.com/paper/2111.05328/) (2021)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Data Augmentation Alone Can Improve Adversarial Training](https://mlsanity.com/paper/2301.09879/) (2023)
- [A Partition Filter Network for Joint Entity and Relation Extraction](https://mlsanity.com/paper/2108.12202/) (2021)
- [T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition](https://mlsanity.com/paper/2209.12616/) (2022)
- [Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention](https://mlsanity.com/paper/1804.06987/) (2018)
- [KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction](https://mlsanity.com/paper/2403.07969/) (2024)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extraction](https://mlsanity.com/paper/2312.15548/) (2023)
- [On Robustness and Bias Analysis of BERT-based Relation Extraction](https://mlsanity.com/paper/2009.06206/) (2020)
- [A Multilingual Evaluation of NER Robustness to Adversarial Inputs](https://mlsanity.com/paper/2305.18933/) (2023)
- [Graph Learning with Loss-Guided Training](https://mlsanity.com/paper/2006.00460/) (2020)
- [Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials](https://mlsanity.com/paper/2402.19076/) (2024)

## Built on by (most cited first, 1 of 1)

- [KnowCoder-V2: Deep Knowledge Analysis](https://mlsanity.com/paper/2506.06881/) (2025)

## Abstract (arXiv)

In this paper, we aim to enhance the robustness of Universal Information Extraction (UIE) by introducing a new benchmark dataset, a comprehensive evaluation, and a feasible solution. Existing robust benchmark datasets have two key limitations: 1) They generate only a limited range of perturbations for a single Information Extraction (IE) task, which fails to evaluate the robustness of UIE models effectively; 2) They rely on small models or handcrafted rules to generate perturbations, often resulting in unnatural adversarial examples. Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench, which utilizes LLMs to generate more diverse and realistic perturbations across different IE tasks. Based on this dataset, we comprehensively evaluate existing UIE models and reveal that both LLM-based models and other models suffer from significant performance drops. To improve robustness and reduce training costs, we propose a data-augmentation solution that dynamically selects hard samples for iterative training based on the model's inference loss. Experimental results show that training with only 15% of the data leads to an average 7.5% relative performance improvement across three IE tasks.