# UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction

arXiv: https://arxiv.org/abs/2303.01194 · HTML: https://mlsanity.com/paper/2303.01194/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [Quantifying Intimacy in Language](https://mlsanity.com/paper/2011.03020/) (2020)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)
- [How to prepare your task head for finetuning](https://mlsanity.com/paper/2302.05779/) (2023)
- [SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis](https://mlsanity.com/paper/2210.01108/) (2022)

## Built on by (most cited first, 8 of 8)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://mlsanity.com/paper/2310.17918/) (2023)
- [The Battle of LLMs: A Comparative Study in Conversational QA Tasks](https://mlsanity.com/paper/2405.18344/) (2024)
- [ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about](https://mlsanity.com/paper/2304.03325/) (2023)
- [Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection](https://mlsanity.com/paper/2405.17964/) (2024)
- [KSHSeek: Data-Driven Approaches to Mitigating and Detecting Knowledge-Shortcut Hallucinations in Generative Models](https://mlsanity.com/paper/2503.19482/) (2025)
- [A Survey on the Real Power of ChatGPT](https://mlsanity.com/paper/2405.00704/) (2024)

## Abstract (arXiv)

This paper describes the submission of UZH_CLyp for the SemEval 2023 Task 9 "Multilingual Tweet Intimacy Analysis". We achieved second-best results in all 10 languages according to the official Pearson's correlation regression evaluation measure. Our cross-lingual transfer learning approach explores the benefits of using a Head-First Fine-Tuning method (HeFiT) that first updates only the regression head parameters and then also updates the pre-trained transformer encoder parameters at a reduced learning rate. Additionally, we study the impact of using a small set of automatically generated examples (in our case, from ChatGPT) for low-resource settings where no human-labeled data is available. Our study shows that HeFiT stabilizes training and consistently improves results for pre-trained models that lack domain adaptation to tweets. Our study also shows a noticeable performance increase in cross-lingual learning when synthetic data is used, confirming the usefulness of current text generation systems to improve zero-shot baseline results. Finally, we examine how possible inconsistencies in the annotated data contribute to cross-lingual interference issues.