# GPT Self-Supervision for a Better Data Annotator

arXiv: https://arxiv.org/abs/2306.04349 · HTML: https://mlsanity.com/paper/2306.04349/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [A Survey on Deep Learning Techniques for Stereo-based Depth Estimation](https://mlsanity.com/paper/2006.02535/) (2020)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [What can Data-Centric AI Learn from Data and ML Engineering?](https://mlsanity.com/paper/2112.06439/) (2021)
- [Learning to Adapt to Light](https://mlsanity.com/paper/2202.08098/) (2022)
- [Semi-Supervised Visual Tracking of Marine Animals using Autonomous Underwater Vehicles](https://mlsanity.com/paper/2302.07344/) (2023)
- [Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations](https://mlsanity.com/paper/2305.14556/) (2023)
- [Dynamic Curriculum Learning for Great Ape Detection in the Wild](https://mlsanity.com/paper/2205.00275/) (2022)
- [Tackling Clutter in Radar Data -- Label Generation and Detection Using PointNet++](https://mlsanity.com/paper/2303.09530/) (2023)
- [Context-Driven Detection of Invertebrate Species in Deep-Sea Video](https://mlsanity.com/paper/2206.00718/) (2022)

## Built on by (most cited first, 5 of 5)

- [Label-free Node Classification on Graphs with Large Language Models (LLMS)](https://mlsanity.com/paper/2310.04668/) (2023)
- [Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy](https://mlsanity.com/paper/2402.16041/) (2024)
- [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://mlsanity.com/paper/2407.04693/) (2024)
- [The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators](https://mlsanity.com/paper/2407.11004/) (2024)
- [How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding](https://mlsanity.com/paper/2402.19116/) (2024)

## Abstract (arXiv)

The task of annotating data into concise summaries poses a significant challenge across various domains, frequently requiring the allocation of significant time and specialized knowledge by human experts. Despite existing efforts to use large language models for annotation tasks, significant problems such as limited applicability to unlabeled data, the absence of self-supervised methods, and the lack of focus on complex structured data still persist. In this work, we propose a GPT self-supervision annotation method, which embodies a generating-recovering paradigm that leverages the one-shot learning capabilities of the Generative Pretrained Transformer (GPT). The proposed approach comprises a one-shot tuning phase followed by a generation phase. In the one-shot tuning phase, we sample a data from the support set as part of the prompt for GPT to generate a textual summary, which is then used to recover the original data. The alignment score between the recovered and original data serves as a self-supervision navigator to refine the process. In the generation stage, the optimally selected one-shot sample serves as a template in the prompt and is applied to generating summaries from challenging datasets. The annotation performance is evaluated by tuning several human feedback reward networks and by calculating alignment scores between original and recovered data at both sentence and structure levels. Our self-supervised annotation method consistently achieves competitive scores, convincingly demonstrating its robust strength in various data-to-summary annotation tasks.