# PESTO: Switching Point based Dynamic and Relative Positional Encoding for Code-Mixed Languages

arXiv: https://arxiv.org/abs/2111.06599 · HTML: https://mlsanity.com/paper/2111.06599/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 5 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020)
- [kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification](https://mlsanity.com/paper/2009.03673/) (2020)

## Built on by (most cited first, 3 of 3)

- [CONFLATOR: Incorporating Switching Point based Rotatory Positional Encodings for Code-Mixed Language Modeling](https://mlsanity.com/paper/2309.05270/) (2023)
- [Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish Memes](https://mlsanity.com/paper/2309.06517/) (2023)
- [CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling](https://mlsanity.com/paper/2505.12587/) (2025)

## Abstract (arXiv)

NLP applications for code-mixed (CM) or mix-lingual text have gained a significant momentum recently, the main reason being the prevalence of language mixing in social media communications in multi-lingual societies like India, Mexico, Europe, parts of USA etc. Word embeddings are basic build-ing blocks of any NLP system today, yet, word embedding for CM languages is an unexplored territory. The major bottleneck for CM word embeddings is switching points, where the language switches. These locations lack in contextually and statistical systems fail to model this phenomena due to high variance in the seen examples. In this paper we present our initial observations on applying switching point based positional encoding techniques for CM language, specifically Hinglish (Hindi - English). Results are only marginally better than SOTA, but it is evident that positional encoding could bean effective way to train position sensitive language models for CM text.