# UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing

arXiv: https://arxiv.org/abs/2404.01253 · HTML: https://mlsanity.com/paper/2404.01253/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View](https://mlsanity.com/paper/2203.12258/) (2022)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts](https://mlsanity.com/paper/2110.07280/) (2021)
- [Assessing the Reliability of Large Language Model Knowledge](https://mlsanity.com/paper/2310.09820/) (2023)
- [The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models](https://mlsanity.com/paper/2311.01307/) (2023)
- [Task-specific Pre-training and Prompt Decomposition for Knowledge Graph Population with Language Models](https://mlsanity.com/paper/2208.12539/) (2022)
- [Towards Alleviating the Object Bias in Prompt Tuning-based Factual Knowledge Extraction](https://mlsanity.com/paper/2306.03378/) (2023)

## Built on by (most cited first, 2 of 2)

- [Evaluating and Improving Graph to Text Generation with Large Language Models](https://mlsanity.com/paper/2501.14497/) (2025)
- [Probing the Knowledge Boundary: An Interactive Agentic Framework for Deep Knowledge Extraction](https://mlsanity.com/paper/2602.00959/) (2026)

## Abstract (arXiv)

Several recent papers have investigated the potential of language models as knowledge bases as well as the existence of severe biases when extracting factual knowledge. In this work, we focus on the factual probing performance over unseen prompts from tuning, and using a probabilistic view we show the inherent misalignment between pre-training and downstream tuning objectives in language models for probing knowledge. We hypothesize that simultaneously debiasing these objectives can be the key to generalisation over unseen prompts. We propose an adapter-based framework, UniArk, for generalised and consistent factual knowledge extraction through simple methods without introducing extra parameters. Extensive experiments show that UniArk can significantly improve the model's out-of-domain generalisation as well as consistency under various prompts. Additionally, we construct ParaTrex, a large-scale and diverse dataset for measuring the inconsistency and out-of-domain generation of models. Further, ParaTrex offers a reference method for constructing paraphrased datasets using large language models.