# Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errors

arXiv: https://arxiv.org/abs/2402.01082 · HTML: https://mlsanity.com/paper/2402.01082/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 68 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Teaching Arithmetic to Small Transformers](https://mlsanity.com/paper/2307.03381/) (2023)
- [Neural GPUs Learn Algorithms](https://mlsanity.com/paper/1511.08228/) (2015)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [SALSA: Attacking Lattice Cryptography with Transformers](https://mlsanity.com/paper/2207.04785/) (2022)
- [Grid Long Short-Term Memory](https://mlsanity.com/paper/1507.01526/) (2015)
- [Learning Simple Algorithms from Examples](https://mlsanity.com/paper/1511.07275/) (2015)
- [SALSA PICANTE: a machine learning attack on LWE with binary secrets](https://mlsanity.com/paper/2303.04178/) (2023)
- [Can Sequence-to-Sequence Models Crack Substitution Ciphers?](https://mlsanity.com/paper/2012.15229/) (2020)
- [Solving Math Word Problems with Double-Decoder Transformer](https://mlsanity.com/paper/1908.10924/) (2019)
- [Solving Arithmetic Word Problems with Transformers and Preprocessing of Problem Text](https://mlsanity.com/paper/2106.00893/) (2021)
- [An efficient algorithm for integer lattice reduction](https://mlsanity.com/paper/2303.02226/) (2023)
- [Learning the Enigma with Recurrent Neural Networks](https://mlsanity.com/paper/1708.07576/) (2017)
- [Using fuzzy bits and neural networks to partially invert few rounds of some cryptographic hash functions](https://mlsanity.com/paper/1901.02438/) (2019)

## Built on by (most cited first, 7 of 7)

- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models](https://mlsanity.com/paper/2411.02083/) (2024)
- [Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmetic](https://mlsanity.com/paper/2410.03569/) (2024)
- [TAPAS: Datasets for Learning the Learning with Errors Problem](https://mlsanity.com/paper/2510.08797/) (2025)
- [Improving ML Attacks on LWE with Data Repetition and Stepwise Regression](https://mlsanity.com/paper/2604.03903/) (2026)
- [Learning Large-Scale Modular Addition with an Auxiliary Modulus](https://mlsanity.com/paper/2605.07648/) (2026)
- [Discovering Lattice Reduction Strategies via Self-Play](https://mlsanity.com/paper/2606.15301/) (2026)

## Abstract (arXiv)

Learning with Errors (LWE) is a hard math problem underlying recently standardized post-quantum cryptography (PQC) systems for key exchange and digital signatures. Prior work proposed new machine learning (ML)-based attacks on LWE problems with small, sparse secrets, but these attacks require millions of LWE samples to train on and take days to recover secrets. We propose three key methods -- better preprocessing, angular embeddings and model pre-training -- to improve these attacks, speeding up preprocessing by 25× and improving model sample efficiency by 10×. We demonstrate for the first time that pre-training improves and reduces the cost of ML attacks on LWE. Our architecture improvements enable scaling to larger-dimension LWE problems: this work is the first instance of ML attacks recovering sparse binary secrets in dimension n=1024, the smallest dimension used in practice for homomorphic encryption applications of LWE where sparse binary secrets are proposed.