# Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC

arXiv: https://arxiv.org/abs/2605.25626 · HTML: https://mlsanity.com/paper/2605.25626/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [Benchmarking Machine Translation with Cultural Awareness](https://mlsanity.com/paper/2305.14328/) (2023)
- [MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning](https://mlsanity.com/paper/2504.10160/) (2025)
- [MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms](https://mlsanity.com/paper/2402.14154/) (2024)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters](https://mlsanity.com/paper/2507.13618/) (2025)
- [SLANG: New Concept Comprehension of Large Language Models](https://mlsanity.com/paper/2401.12585/) (2024)
- [MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts](https://mlsanity.com/paper/2406.12549/) (2024)
- [TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds](https://mlsanity.com/paper/2403.12410/) (2024)
- [Redefining Machine Translation on Social Network Services with Large Language Models](https://mlsanity.com/paper/2504.07901/) (2025)
- [MT3: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2505.19714/) (2025)
- [PopSim: Social Network Simulation for Social Media Popularity Prediction](https://mlsanity.com/paper/2512.02533/) (2025)
- [DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation](https://mlsanity.com/paper/2510.09116/) (2025)
- [SlangDIT: Benchmarking LLMs in Interpretative Slang Translation](https://mlsanity.com/paper/2505.14181/) (2025)
- [Can Large Language Models Understand Internet Buzzwords Through User-Generated Content](https://mlsanity.com/paper/2505.15071/) (2025)
- [Cross-Platform Short-Video Diplomacy: Topic and Sentiment Analysis of China-US Relations on Douyin and TikTok](https://mlsanity.com/paper/2510.22415/) (2025)
- [Contrastive Learning of Sociopragmatic Meaning in Social Media](https://mlsanity.com/paper/2203.07648/) (2022)
- [TransBench: Benchmarking Machine Translation for Industrial-Scale Applications](https://mlsanity.com/paper/2505.14244/) (2025)
- [RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services](https://mlsanity.com/paper/2511.07070/) (2025)
- [X-MuTeST: A Multilingual Benchmark for Explainable Hate Speech Detection and A Novel LLM-consulted Explanation Framework](https://mlsanity.com/paper/2601.03194/) (2026)

## Built on by (most cited first, 2 of 2)

- [When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation](https://mlsanity.com/paper/2609.08156/) (2026)
- [Evaluating Communicative Success in Machine-Translated Conversation](https://mlsanity.com/paper/2609.19885/) (2026)

## Abstract (arXiv)

Social media platforms enable large-scale cross-lingual communication, but translating user-generated content (UGC) remains challenging due to its informal style, cultural references, and interaction-based expressions. While recent LLMs have improved translation quality, existing benchmarks and metrics often fail to capture whether translations convey intended meaning and cultural resonance in real-world settings. In this work, we introduce CULTURE-MT, a benchmark for social media translation that focuses on both CULtural Transmission and UGC-specific emotion REsonance. CULTURE-MT consists of 1,002 UGC notes across 14 domains, categorized into four types based on culture-loaded symbols and linguistic style features. We also construct UGC-oriented training data to fine-tune Qwen3-8B and Qwen3-32B as baselines. We propose cultural effectiveness as a new evaluation criterion, focusing on expression accuracy and cultural adaptability. Testing 15 models, including the baselines, we find that traditional metrics fail to capture cultural effectiveness. We also observe that cultural effectiveness on base LLMs correlates with model size. Our work provides a comprehensive evaluation system for UGC translation models and will offer an open evaluation platform to advance research in this area. We release the CULTURE-MT benchmark and provide an online leaderboard where submitted translation results can be evaluated by our trained JUDGER.