# Monotonic Representation of Numeric Properties in Language Models

arXiv: https://arxiv.org/abs/2403.10381 · HTML: https://mlsanity.com/paper/2403.10381/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 70 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Inspecting and Editing Knowledge Representations in Language Models](https://mlsanity.com/paper/2304.00740/) (2023)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)

## Built on by (most cited first, 15 of 15)

- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [When Models Manipulate Manifolds: The Geometry of a Counting Task](https://mlsanity.com/paper/2601.04480/) (2026)
- [Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models](https://mlsanity.com/paper/2403.19521/) (2024)
- [All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling](https://mlsanity.com/paper/2410.23501/) (2024)
- [Feature contamination: Neural networks learn uncorrelated features and fail to generalize](https://mlsanity.com/paper/2406.03345/) (2024)
- [Low-Rank Adapting Models for Sparse Autoencoders](https://mlsanity.com/paper/2501.19406/) (2025)
- [Investigating Representation Universality: Case Study on Genealogical Representations](https://mlsanity.com/paper/2410.08255/) (2024)
- [Harmonic Loss Trains Interpretable AI Models](https://mlsanity.com/paper/2502.01628/) (2025)
- [Medical Interpretability and Knowledge Maps of Large Language Models](https://mlsanity.com/paper/2510.11390/) (2025)
- [A Matter of Time: Revealing the Structure of Time in Vision-Language Models](https://mlsanity.com/paper/2510.19559/) (2025)
- [Compositional Literary Primitives in Instruction-Tuned LLMs: Cross-Architectural SAE Features for Self, Style, and Affect](https://mlsanity.com/paper/2605.18808/) (2026)
- [Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detection](https://mlsanity.com/paper/2606.21078/) (2026)

## Abstract (arXiv)

Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model's internal representations is not understood well. Here, we introduce a simple method for finding and editing representations of numeric properties such as an entity's birth year. Empirically, we find low-dimensional subspaces that encode numeric properties monotonically, in an interpretable and editable fashion. When editing representations along directions in these subspaces, LM output changes accordingly. For example, by patching activations along a "birthyear" direction we can make the LM express an increasingly late birthyear: Karl Popper was born in 1929, Karl Popper was born in 1957, Karl Popper was born in 1968. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining. Code: https://github.com/bheinzerling/numeric-property-repr