Linear Regression

Linear Regression

What is Linear Regression?

Linear regression models the relationship between one or more explanatory variables (predictors) and a continuous outcome variable. It assumes the relationship is roughly a straight line.

Real example: Predict a house’s sale price (y) from its size in square feet (x).

Retrieval prompt: Before reading on, try to name two real-world pairs of variables you think might have a roughly linear relationship.

The Model

The simple linear regression model is:

y = β₀ + β₁x + ε

Generation prompt: Before reading the next section — if you had to draw a line through a scatter plot of points, how would you decide which line is “best”? Write down your criterion.

How do we find the best line? — Ordinary Least Squares (OLS)

The “best” line minimizes the sum of squared residuals (SSR):

minimize  Σ (yᵢ − ŷᵢ)²

Where ŷᵢ = β₀ + β₁xᵢ is the predicted value and (yᵢ − ŷᵢ) is the residual — the vertical distance from the point to the line.

We square residuals so positive and negative errors don’t cancel out, and larger errors are penalised more.

The OLS solution (derived via calculus — set partial derivatives to zero) gives closed-form formulas¹:

β₁ = Σ((xᵢ − x̄)(yᵢ − ȳ)) / Σ((xᵢ − x̄)²)
β₀ = ȳ − β₁x̄

Where x̄ and ȳ are the sample means of x and y.

Intuition: The slope β₁ is the covariance of x and y divided by the variance of x. If x and y move together, β₁ is large.

Interpreting the output

Slope (β₁)

“The predicted y changes by β₁ units for every 1-unit increase in x.”

If β₁ = 150, each additional square foot adds $150 to predicted price.

Intercept (β₀)

“The predicted y when x = 0.” Often not meaningful on its own (what is a 0 sq ft house?), but needed mathematically.

R² (Coefficient of Determination)

Proportion of variance in y that the model explains:

R² = 1 − (SS_residual / SS_total)

• R² = 0: The model explains nothing (same as using the mean).
• R² = 1: The model explains everything (all points on the line).
• R² = 0.6: 60% of the variance in y is explained by x.

Warning: R² always increases when you add more predictors, even useless ones. Adjusted R² penalises extra variables².

Assumptions of Linear Regression

These matter for valid inference (p-values, confidence intervals). For working-knowledge, remember the acronym LINE³:

If assumptions are badly violated, predictions may still be okay, but your p-values and confidence intervals won’t be trustworthy².

Worked Example

Exercise: Do this by hand before checking the answer.

Data: Hours studied (x) vs exam score (y)

Step 1: Compute x̄ = 3, ȳ = 60.

Step 2: Compute β₁.

• Numerator: Σ((xᵢ − x̄)(yᵢ − ȳ)) = (−2)(−20) + (−1)(−10) + (0)(0) + (1)(10) + (2)(20) = 40 + 10 + 0 + 10 + 40 = 100
• Denominator: Σ((xᵢ − x̄)²) = 4 + 1 + 0 + 1 + 4 = 10
• β₁ = 100 / 10 = 10

Step 3: Compute β₀ = ȳ − β₁x̄ = 60 − (10)(3) = 30

Model: score = 30 + 10 × hours

Each additional hour studied predicts 10 more points on the exam.

Step 4: R²

• SS_total = Σ(yᵢ − ȳ)² = (−20)² + (−10)² + 0² + 10² + 20² = 1000
• SS_residual = Σ(yᵢ − ŷᵢ)² → ŷ: [40, 50, 60, 70, 80] → residuals are all 0 → SS_residual = 0
• R² = 1 − 0/1000 = 1.0 (perfect fit — the data is perfectly linear on purpose)

Quick Python Demo

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3], [4], [5]])  # predictor
y = np.array([40, 50, 60, 70, 80])       # target

model = LinearRegression()
model.fit(X, y)

print(f"Intercept: {model.intercept_:.2f}")   # 30.00
print(f"Slope:     {model.coef_[0]:.2f}")      # 10.00
print(f"R²:        {model.score(X, y):.2f}")   # 1.00

Key Takeaways

• Linear regression models a continuous outcome as a linear combination of predictors.
• OLS finds the line that minimises sum of squared residuals.
• β₁ tells you the expected change in y per unit change in x.
• R² tells you how much variance the model explains (0–1 scale).
• LINE assumptions (Linearity, Independence, Normality, Equal variance) matter for inference.
• The model is just y = β₀ + β₁x + ε — simple but powerful.

Final retrieval prompt: Cover the page and try to write the formulas for β₀, β₁, and R² from memory. Then check. Which assumption would be violated if residuals formed a fan shape?