Linear regression, low-level

Linear regression is the classic entry point to deep learning: a single neuron, no activation function, trained by gradient descent. Understanding how it works mechanically — not just model.fit() — lays the groundwork for everything that follows (deep networks, backprop, optimizers).

The model

We look for an affine function that approximates $y$ from $x$ :

$$\hat{y} = w x + b$$

With several input variables, $x \in \mathbb{R}^n$ becomes a vector, and so does $w \in \mathbb{R}^n$ :

$$\hat{y} = w^\top x + b = \sum_{i=1}^{n} w_i x_i + b$$

The cost function

We measure the error over the whole training set with the mean squared error:

$$J(w, b) = \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)^2$$

where $m$ is the number of examples. Squaring penalises large errors more heavily and makes the function differentiable everywhere, unlike the absolute value.

The gradient

To minimise $J$ , we compute its derivative with respect to each parameter. For $w$ :

$$\frac{\partial J}{\partial w} = \frac{2}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right) x^{(i)}$$

And for $b$ :

$$\frac{\partial J}{\partial b} = \frac{2}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)$$

Gradient descent

We update the parameters in the direction opposite to the gradient, scaled by a learning rate $\eta$ :

$$w \leftarrow w - \eta \frac{\partial J}{\partial w}$$

$$b \leftarrow b - \eta \frac{\partial J}{\partial b}$$

We repeat until convergence. An $\eta$ that is too large makes training diverge; too small, and it converges, but very slowly.

Minimal implementation

import numpy as np

def train(X, y, lr=0.01, epochs=1000):
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0

    for _ in range(epochs):
        y_hat = X @ w + b
        error = y_hat - y

        dw = (2 / m) * X.T @ error
        db = (2 / m) * np.sum(error)

        w -= lr * dw
        b -= lr * db

    return w, b

The analytical solution

For this specific case, there is an exact solution, with no iteration: the normal equations.

$$w = (X^\top X)^{-1} X^\top y$$

It avoids gradient descent entirely, but becomes expensive when $n$ is large (matrix inversion is $O(n^3)$ ), or impossible if $X^\top X$ is not invertible. Gradient descent, on the other hand, generalises directly to deep networks — which is why we learn it first.