# Generating Synthetic Oracle Datasets to Analyze Noise Impact: A Study on Building Function Classification Using Tweets

arXiv: https://arxiv.org/abs/2503.22856 · HTML: https://mlsanity.com/paper/2503.22856/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning](https://mlsanity.com/paper/2405.02501/) (2024)
- [Changes in Twitter geolocations: Insights and suggestions for future usage](https://mlsanity.com/paper/2108.12251/) (2021)
- [Where did you tweet from? Inferring the origin locations of tweets based on contextual information](https://mlsanity.com/paper/2211.16506/) (2022)

## Built on by (most cited first, 2 of 2)

- [CoGraM: Context-sensitive granular optimization method with rollback for robust model fusion](https://mlsanity.com/paper/2512.03610/) (2025)
- [Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasets](https://mlsanity.com/paper/2603.13625/) (2026)

## Abstract (arXiv)

Tweets provides valuable semantic context for earth observation tasks and serves as a complementary modality to remote sensing imagery. In building function classification (BFC), tweets are often collected using geographic heuristics and labeled via external databases, an inherently weakly supervised process that introduces both label noise and sentence level feature noise (e.g., irrelevant or uninformative tweets). While label noise has been widely studied, the impact of sentence level feature noise remains underexplored, largely due to the lack of clean benchmark datasets for controlled analysis. In this work, we propose a method for generating a synthetic oracle dataset using LLM, designed to contain only tweets that are both correctly labeled and semantically relevant to their associated buildings. This oracle dataset enables systematic investigation of noise impacts that are otherwise difficult to isolate in real-world data. To assess its utility, we compare model performance using Naive Bayes and mBERT classifiers under three configurations: real vs. synthetic training data, and cross-domain generalization. Results show that noise in real tweets significantly degrades the contextual learning capacity of mBERT, reducing its performance to that of a simple keyword-based model. In contrast, the clean synthetic dataset allows mBERT to learn effectively, outperforming Naive Bayes Bayes by a large margin. These findings highlight that addressing feature noise is more critical than model complexity in this task. Our synthetic dataset offers a novel experimental environment for future noise injection studies and is publicly available on GitHub.