# Differentially Private Ordinary Least Squares

arXiv: https://arxiv.org/abs/1507.02482 · HTML: https://mlsanity.com/paper/1507.02482/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2015.

## Built on by (most cited first, 30 of 36)

- [Differentially private inference via noisy optimization](https://mlsanity.com/paper/2103.11003/) (2021)
- [Optimal Learners for Realizable Regression: PAC Learning and Online Learning](https://mlsanity.com/paper/2307.03848/) (2023)
- [Private Prediction Sets](https://mlsanity.com/paper/2102.06202/) (2021)
- [Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation](https://mlsanity.com/paper/2402.13531/) (2024)
- [An Introduction to Johnson-Lindenstrauss Transforms](https://mlsanity.com/paper/2103.00564/) (2021)
- [Differentially Private Regression with Unbounded Covariates](https://mlsanity.com/paper/2202.11199/) (2022)
- [Differentially Private Bootstrap: New Privacy Analysis and Inference Strategies](https://mlsanity.com/paper/2210.06140/) (2022)
- [The Test of Tests: A Framework For Differentially Private Hypothesis Testing](https://mlsanity.com/paper/2302.04260/) (2023)
- [Insufficient Statistics Perturbation: Stable Estimators for Private Least Squares](https://mlsanity.com/paper/2404.15409/) (2024)
- [Improved Differentially Private Regression via Gradient Boosting](https://mlsanity.com/paper/2303.03451/) (2023)
- [Robust Testing and Estimation under Manipulation Attacks](https://mlsanity.com/paper/2104.10740/) (2021)
- [Resampling methods for private statistical inference](https://mlsanity.com/paper/2402.07131/) (2024)
- [Private Regression via Data-Dependent Sufficient Statistic Perturbation](https://mlsanity.com/paper/2405.15002/) (2024)
- [Brownian Noise Reduction: Maximizing Privacy Subject to Accuracy Constraints](https://mlsanity.com/paper/2206.07234/) (2022)
- [Sample-Optimal Private Regression in Polynomial Time](https://mlsanity.com/paper/2503.24321/) (2025)
- [PrivATE: Differentially Private Confidence Intervals for Average Treatment Effects](https://mlsanity.com/paper/2505.21641/) (2025)
- [The Gaussian Mixing Mechanism: Renyi Differential Privacy via Gaussian Sketches](https://mlsanity.com/paper/2505.24603/) (2025)
- [Statistical Inference for Differentially Private Stochastic Gradient Descent](https://mlsanity.com/paper/2507.20560/) (2025)
- [Differentially Private Variable Selection via the Knockoff Filter](https://mlsanity.com/paper/2109.05402/) (2021)
- [Label Differential Privacy via Aggregation](https://mlsanity.com/paper/2310.10092/) (2023)
- [Private Statistical Estimation via Truncation](https://mlsanity.com/paper/2505.12541/) (2025)
- [High-Dimensional Differentially Private Quantile Regression: Distributed Estimation and Statistical Inference](https://mlsanity.com/paper/2508.05212/) (2025)
- [Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees](https://mlsanity.com/paper/2510.16974/) (2025)
- [Near-Optimal Private Linear Regression via Iterative Hessian Mixing](https://mlsanity.com/paper/2601.07545/) (2026)
- [Linear Model Against Malicious Adversaries with Local Differential Privacy](https://mlsanity.com/paper/2202.02448/) (2022)
- [Differentially Private Multi-Party Data Release for Linear Regression](https://mlsanity.com/paper/2206.07998/) (2022)
- [Large-Scale Public Data Improves Differentially Private Image Generation Quality](https://mlsanity.com/paper/2309.00008/) (2023)
- [The Normal Distributions Indistinguishability Spectrum and its Application to Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2309.01243/) (2023)
- [Differentially Private Geodesic Regression](https://mlsanity.com/paper/2504.11304/) (2025)
- [Enhancing Differentially Private Linear Regression via Public Second-Moment](https://mlsanity.com/paper/2508.18037/) (2025)

## Abstract (arXiv)

Linear regression is one of the most prevalent techniques in machine learning, however, it is also common to use linear regression for its explanatory capabilities rather than label prediction. Ordinary Least Squares (OLS) is often used in statistics to establish a correlation between an attribute (e.g. gender) and a label (e.g. income) in the presence of other (potentially correlated) features. OLS assumes a particular model that randomly generates the data, and derives t-values --- representing the likelihood of each real value to be the true correlation. Using t-values, OLS can release a confidence interval, which is an interval on the reals that is likely to contain the true correlation, and when this interval does not intersect the origin, we can reject the null hypothesis as it is likely that the true correlation is non-zero. Our work aims at achieving similar guarantees on data under differentially private estimators. First, we show that for well-spread data, the Gaussian Johnson-Lindenstrauss Transform (JLT) gives a very good approximation of t-values, secondly, when JLT approximates Ridge regression (linear regression with l2-regularization) we derive, under certain conditions, confidence intervals using the projected data, lastly, we derive, under different conditions, confidence intervals for the "Analyze Gauss" algorithm (Dwork et al, STOC 2014).