# Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing

arXiv: https://arxiv.org/abs/2510.08111 · HTML: https://mlsanity.com/paper/2510.08111/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 27 of 27)

- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://mlsanity.com/paper/2304.10703/) (2023)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Natural Language Processing in the Legal Domain](https://mlsanity.com/paper/2302.12039/) (2023)
- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [BingoGuard: LLM Content Moderation Tools with Risk Levels](https://mlsanity.com/paper/2503.06550/) (2025)
- [LLMs Provide Unstable Answers to Legal Questions](https://mlsanity.com/paper/2502.05196/) (2025)
- [From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification](https://mlsanity.com/paper/2310.11878/) (2023)
- [An Empirical Study of Affiliate Marketing Disclosures on YouTube and Pinterest](https://mlsanity.com/paper/1803.08488/) (2018)
- [BLT: Can Large Language Models Handle Basic Legal Text?](https://mlsanity.com/paper/2311.09693/) (2023)
- [Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media](https://mlsanity.com/paper/2306.05115/) (2023)
- [Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning](https://mlsanity.com/paper/2502.05675/) (2025)
- [LExT: Towards Evaluating Trustworthiness of Natural Language Explanations](https://mlsanity.com/paper/2504.06227/) (2025)
- [Characterising and Detecting Sponsored Influencer Posts on Instagram](https://mlsanity.com/paper/2011.05757/) (2020)
- [Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Base](https://mlsanity.com/paper/2406.13217/) (2024)
- [Across Platforms and Languages: Dutch Influencers and Legal Disclosures on Instagram, YouTube and TikTok](https://mlsanity.com/paper/2407.12451/) (2024)
- [InstaSynth: Opportunities and Challenges in Generating Synthetic Instagram Data with ChatGPT for Sponsored Content Detection](https://mlsanity.com/paper/2403.15214/) (2024)
- [Influencer Self-Disclosure Practices on Instagram: A Multi-Country Longitudinal Study](https://mlsanity.com/paper/2407.09202/) (2024)
- [Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences](https://mlsanity.com/paper/2410.00873/) (2024)
- [Leveraging ChatGPT for Sponsored Ad Detection and Keyword Extraction in YouTube Videos](https://mlsanity.com/paper/2502.15102/) (2025)

## Built on by (most cited first, 4 of 4)

- [Knowing the Form, Not the Function: Automatically Auditing Answer--Authority Decoupling in Legal Benchmarks](https://mlsanity.com/paper/2608.02621/) (2026)
- [ChildSafeAds Shared Task 2026: Commercial Content in Child-Facing YouTube Videos](https://mlsanity.com/paper/2608.19165/) (2026)
- [Cross-Lingual Legal QA for Vietnamese Labour Law: Retrieval, Translation, and Verifier-Guided Correction](https://mlsanity.com/paper/2609.27376/) (2026)
- [N"urnberg NLP at ChildSafeAds 2026: Structurally Dissimilar Voter Ensembles under Four Levels of Data Access](https://mlsanity.com/paper/2609.34986/) (2026)

## Abstract (arXiv)

The rise of influencer marketing has blurred boundaries between organic content and sponsored content, making the enforcement of legal rules relating to transparency challenging. Effective regulation requires applying legal knowledge with a clear purpose and reason, yet current detection methods of undisclosed sponsored content generally lack legal grounding or operate as opaque "black boxes". Using 1,143 Instagram posts, we compare gpt-5-nano and gemini-2.5-flash-lite under three prompting strategies with controlled levels of legal knowledge provided. Both models perform strongly in classifying content as sponsored or not (F1 up to 0.93), though performance drops by over 10 points on ambiguous cases. We further develop a taxonomy of reasoning errors, showing frequent citation omissions (28.57%), unclear references (20.71%), and hidden ads exhibiting the highest miscue rate (28.57%). While adding regulatory text to the prompt improves explanation quality, it does not consistently improve detection accuracy. The contribution of this paper is threefold. First, it makes a novel addition to regulatory compliance technology by providing a taxonomy of common errors in LLM-generated legal reasoning to evaluate whether automated moderation is not only accurate but also legally robust, thereby advancing the transparent detection of influencer marketing content. Second, it features an original dataset of LLM explanations annotated by two students who were trained in influencer marketing law. Third, it combines quantitative and qualitative evaluation strategies for LLM explanations and critically reflects on how these findings can support advertising regulatory bodies in automating moderation processes on a solid legal foundation.