# Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulation

arXiv: https://arxiv.org/abs/2409.19308 · HTML: https://mlsanity.com/paper/2409.19308/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://mlsanity.com/paper/2109.05687/) (2021)
- [Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data](https://mlsanity.com/paper/2009.09139/) (2020)
- [Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning](https://mlsanity.com/paper/2012.02462/) (2020)
- [Rethinking Learning Rate Tuning in the Era of Large Language Models](https://mlsanity.com/paper/2309.08859/) (2023)
- [Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models](https://mlsanity.com/paper/2407.16470/) (2024)
- [Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias](https://mlsanity.com/paper/2311.00217/) (2023)
- [Towards Data-efficient Customer Intent Recognition with Prompt-based Learning Paradigm](https://mlsanity.com/paper/2309.14779/) (2023)
- [Optimizing Multi-Domain Performance with Active Learning-based Improvement Strategies](https://mlsanity.com/paper/2304.06277/) (2023)

## Built on by (most cited first, 4 of 4)

- [Generative AI in Transportation Planning: A Survey](https://mlsanity.com/paper/2503.07158/) (2025)
- [Beyond Automation: Rethinking Work, Creativity, and Governance in the Age of Generative AI](https://mlsanity.com/paper/2512.11893/) (2025)
- [Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content](https://mlsanity.com/paper/2602.19177/) (2026)
- [The Rise of AI in Weather and Climate Information and its Impact on Global Inequality](https://mlsanity.com/paper/2603.05710/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have transformed natural language processing, yet face challenges in specialized tasks such as simulating opinions on environmental policies. This paper introduces a novel fine-tuning approach that integrates socio-demographic data from the UK Household Longitudinal Study, uniquely using profiling factors, such as age, gender, income, education, and region. This method enhances the accuracy and representation of generated views. By emulating diverse synthetic profiles, the fine-tuned models significantly outperform pre-trained counterparts, achieving measurable improvements in capturing demographic nuances. Evaluation metrics, including Chi-Squared, Cosine Similarity, Jaccard Index, and KL-divergence, reveal a strong alignment between synthetic and real-world opinions. This work demonstrates the potential of fine-tuned LLMs tailored to societal contexts to enable more ethical and precise policy simulations. Its broader implications include deploying LLMs in domains like healthcare and education, fostering inclusive and data-driven decision-making in both research and practice.