# CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback

arXiv: https://arxiv.org/abs/2411.09073 · HTML: https://mlsanity.com/paper/2411.09073/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 67 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://mlsanity.com/paper/2401.06080/) (2024)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Self-Boosting Large Language Models with Synthetic Preference Data](https://mlsanity.com/paper/2410.06961/) (2024)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text](https://mlsanity.com/paper/2107.06483/) (2021)
- [HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text](https://mlsanity.com/paper/2107.03760/) (2021)
- [Can ChatGPT be used to generate scientific hypotheses?](https://mlsanity.com/paper/2304.12208/) (2023)
- [Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text](https://mlsanity.com/paper/2106.10123/) (2021)
- [MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences](https://mlsanity.com/paper/2410.02381/) (2024)
- [Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution](https://mlsanity.com/paper/2402.11525/) (2024)
- [ALaRM: Align Language Models via Hierarchical Rewards Modeling](https://mlsanity.com/paper/2403.06754/) (2024)
- [Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities](https://mlsanity.com/paper/2410.11079/) (2024)
- [Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentences](https://mlsanity.com/paper/2410.10580/) (2024)

## Built on by (most cited first, 5 of 5)

- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs](https://mlsanity.com/paper/2510.05678/) (2025)
- [The Hardness of Achieving Impact in AI for Social Impact Research: A Ground-Level View of Challenges & Opportunities](https://mlsanity.com/paper/2506.14829/) (2025)
- [Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment](https://mlsanity.com/paper/2511.10670/) (2025)
- [Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMs](https://mlsanity.com/paper/2602.11181/) (2026)

## Abstract (arXiv)

Large language models (LLMs) show strong performance across many tasks but remain weak at understanding code-mixed (CM) language. Despite this limitation, improving LLMs for CM tasks has received little attention. To address this gap, we propose CHAI, a general-purpose framework for enhancing LLM performance on CM tasks, focusing on CM translation. CHAI leverages four key ideas. First, we investigate the use of LLMs as annotators to address the scarcity of high-quality CM datasets. Second, we leverage these LLM annotations to generate large-scale preference data and apply reinforcement learning from AI feedback (RLAIF) to improve CM translation. Third, we incorporate LLM-generated domain knowledge as a constitution, enabling iterative response refinement. Fourth, we perform extensive evaluations on real-world datasets and settings. Results show that CHAI-powered LLMs outperform state-of-the-art open-source models by 68.45% on average in human-adjudicated win rate on CM translation tasks. This work is a step toward more inclusive open-source code-mixed LLMs.