# The Problem of Alignment

arXiv: https://arxiv.org/abs/2401.00210 · HTML: https://mlsanity.com/paper/2401.00210/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 11 of its 54 references that are in the map.
- First posted on arXiv 30 Dec 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [Could a Large Language Model be Conscious?](https://mlsanity.com/paper/2303.07103/) (2023)
- [Neuro-Symbolic Artificial Intelligence: Current Trends](https://mlsanity.com/paper/2105.05330/) (2021)
- [Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer](https://mlsanity.com/paper/2310.09762/) (2023)
- [Stochastic LLMs do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMs](https://mlsanity.com/paper/2309.05918/) (2023)
- [Truth Machines: Synthesizing Veracity in AI Language Models](https://mlsanity.com/paper/2301.12066/) (2023)
- [Structured Like a Language Model: Analysing AI as an Automated Subject](https://mlsanity.com/paper/2212.05058/) (2022)

## Built on by (most cited first, 1 of 1)

- [Perception Gaps in Risk, Benefit, and Value Between Experts and Public Challenge Socially Accepted AI](https://mlsanity.com/paper/2412.01459/) (2024)

## Abstract (arXiv)

Large Language Models produce sequences learned as statistical patterns from large corpora. In order not to reproduce corpus biases, after initial training models must be aligned with human values, preferencing certain continuations over others. Alignment, which can be viewed as the superimposition of normative structure onto a statistical model, reveals a conflicted and complex interrelationship between language and technology. This relationship shapes theories of language, linguistic practice and subjectivity, which are especially relevant to the current sophistication in artificially produced text. We examine this practice of structuration as a two-way interaction between users and models by analysing how ChatGPT4 redacts perceived `anomalous' language in fragments of Joyce's Ulysses and the new linguistic practice of prompt engineering. We then situate this alignment problem historically, revisiting earlier postwar linguistic debates which counterposed two views of meaning: as discrete structures, and as continuous probability distributions. We discuss the largely occluded work of the Moscow Linguistic School, which sought to reconcile this opposition. Our attention to the Moscow School and later related arguments by Searle and Kristeva casts the problem of alignment in a new light: as one involving attention to the social structuration of linguistic practice, including structuration of anomalies that, like the Joycean text, exist in defiance of expressive conventions. These debates around the communicative orientation toward language can help explain some of the contemporary behaviours and interdependencies that take place between users and LLMs.