# CONFLATOR: Incorporating Switching Point based Rotatory Positional Encodings for Code-Mixed Language Modeling

arXiv: https://arxiv.org/abs/2309.05270 · HTML: https://mlsanity.com/paper/2309.05270/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text](https://mlsanity.com/paper/2006.00206/) (2020)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [Sentiment Analysis of Code-Mixed Indian Languages: An Overview of SAIL_Code-Mixed Shared Task @ICON-2017](https://mlsanity.com/paper/1803.06745/) (2018)
- [Hater-O-Genius Aggression Classification using Capsule Networks](https://mlsanity.com/paper/2105.11219/) (2021)
- [Understanding Chat Messages for Sticker Recommendation in Messaging Apps](https://mlsanity.com/paper/1902.02704/) (2019)
- [Offense Detection in Dravidian Languages using Code-Mixing Index based Focal Loss](https://mlsanity.com/paper/2111.06916/) (2021)
- [HIT: A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation](https://mlsanity.com/paper/2105.14600/) (2021)
- [An Ensemble Model for Sentiment Analysis of Hindi-English Code-Mixed Data](https://mlsanity.com/paper/1806.04450/) (2018)
- [PESTO: Switching Point based Dynamic and Relative Positional Encoding for Code-Mixed Languages](https://mlsanity.com/paper/2111.06599/) (2021)

## Built on by (most cited first, 2 of 2)

- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling](https://mlsanity.com/paper/2505.12587/) (2025)

## Abstract (arXiv)

The mixing of two or more languages is called Code-Mixing (CM). CM is a social norm in multilingual societies. Neural Language Models (NLMs) like transformers have been effective on many NLP tasks. However, NLM for CM is an under-explored area. Though transformers are capable and powerful, they cannot always encode positional information since they are non-recurrent. Therefore, to enrich word information and incorporate positional information, positional encoding is defined. We hypothesize that Switching Points (SPs), i.e., junctions in the text where the language switches (L1 -> L2 or L2 -> L1), pose a challenge for CM Language Models (LMs), and hence give special emphasis to SPs in the modeling process. We experiment with several positional encoding mechanisms and show that rotatory positional encodings along with switching point information yield the best results. We introduce CONFLATOR: a neural language modeling approach for code-mixed languages. CONFLATOR tries to learn to emphasize switching points using smarter positional encoding, both at unigram and bigram levels. CONFLATOR outperforms the state-of-the-art on two tasks based on code-mixed Hindi and English (Hinglish): (i) sentiment analysis and (ii) machine translation.