Training ML from scratch. Karpathy style

Introduction

The shower in an unfamiliar bathroom

Two taps . One thing you can feel.

You step in. The water is cold.

Nobody gives you a schematic of the valve behind the tiles. Nobody tells you how far a turn goes. You have one instrument, your skin, and one move: nudge a tap, feel the water, keep the change if it helped.

A minute later you are comfortable. You never learned anything about the valve.

That minute is gradient descent converging. The taps are the parameters. The gap between the water and what you wanted is the loss. The nudge is the gradient.

The taps are not equally powerful. And overcorrecting gets you scalded, then frozen, then worse.

What this post builds

You won't probably have to write this. But understanding it will make you a better ML practitioner.

We will build the following in the code:

• The guess. Input in, number out.
• The score. Many mistakes, one number.
• Which way to turn. Nudge a tap, watch the score.
• How far to turn. The setting that decides whether learning works.
• Doing it again. Why it all sits in a loop.

The setup

What we are predicting?

Assume the weights w and bias b are the taps.

The data

y is the water temperature and x is the input water temperature and we have the following data points:

x = [1, 2, 3, 4, 5, 6, 7, 8]
y = [8, 11, 14, 17, 20, 23, 26, 29]

The rule behind those numbers is y = 3x + 5. So you need to guess 3 and 5 correctly. How would you do it?

Eight points, one line. Every point sits exactly on y = 3x + 5. The model can fit this perfectly, which is the point.

The guess

The forward pass

Input in. Number out.

def forward_pass(x, w, b):
    return w * x + b

That is the model. Two numbers it can change, w and b

The score

Many mistakes, one number

You start with a guess and then compare. The difference between them is the error.

Loss is the summation of all training errors as determined by the average of the mean squared error

• y - y_hat One error per example. Some positive, some negative.
• (...) ** 2 Squaring kills the signs, so errors cannot cancel. It also punishes big mistakes harder. An error of 4 is four times worse than an error of 2, not twice.
• np.mean(...) The score means the same thing for 8 examples or 8000.

def compute_loss(y, y_hat):
    return np.mean((y - y_hat) ** 2)

Which way to turn

Nudge it and see

Which way to turn?

Turn w by a tiny amount, eps. Read the loss again and measure the change in loss relative to eps

That is the gradient. Its sign is the direction. Its size is the steepness. It tap has its own gradient i.e. each tap has a different impact on the final water temperature.

The nudge, drawn. Hold every tap still but one and the loss is a curve. Step sideways by eps, measure the drop, divide. Positive slope means opening that tap makes things worse.

def backward_pass(x, y, w, b, eps=1e-3):
    loss   = compute_loss(y, forward_pass(x, w, b))
    loss_w = compute_loss(y, forward_pass(x, w + eps, b))
    loss_b = compute_loss(y, forward_pass(x, w, b + eps))

    g_w = (loss_w - loss) / eps
    g_b = (loss_b - loss) / eps
    return g_w, g_b

You already do this in a shower. Move both taps at once and you learn nothing about which one mattered.

How far to turn

The update, and the one number you tune forever

Direction is useless without step size.

At the start the gradient for w is -198. Turning a tap by 198 of anything takes you straight past comfortable. So scale it down first. That factor is the learning rate.

def update_weights(w, b, g_w, g_b, lr=0.01):
    return w - lr * g_w, b - lr * g_b

The minus sign is the whole idea. Positive gradient, subtract, the parameter falls. Negative gradient, subtract a negative, the parameter rises

The learning rate is the most consequential number in the file

Doing it again

The training loop

Descent is the same small step many times

The gradient is only true where you stand. Take a step and it has changed. So measure again. Each pass is an epoch.

def train(x, y, w=0.0, b=0.0, epochs=2000):
    history = []
    for _ in range(epochs):
        g_w, g_b = backward_pass(x, y, w, b)
        w, b     = update_weights(w, b, g_w, g_b)
        history.append(compute_loss(y, forward_pass(x, w, b)))
    return w, b, history

Order matters. Gradients come before the update. Update first and you have measured a place you already left. It runs without error and learns badly.

The build

End to End Code

Combining all steps together

The data is y = 3x + 5 over eight points. We already know the answer, which is why it is a good test. If w does not land near 3, the bug is in the loop.

import numpy as np


# 1. Guess: turn inputs into predictions
def forward_pass(x, w, b):
    return w * x + b


# 2. Score: one number for how wrong the guesses are
def compute_loss(y, y_hat):
    return np.mean((y - y_hat) ** 2)


# 3. Direction: nudge each parameter, watch the loss move
def backward_pass(x, y, w, b, eps=1e-3):
    loss   = compute_loss(y, forward_pass(x, w, b))
    loss_w = compute_loss(y, forward_pass(x, w + eps, b))
    loss_b = compute_loss(y, forward_pass(x, w, b + eps))
    return (loss_w - loss) / eps, (loss_b - loss) / eps


# 4. Step: move each parameter a little way downhill
def update_weights(w, b, g_w, g_b, lr=0.01):
    return w - lr * g_w, b - lr * g_b


# 5. Repeat
def train(x, y, w=0.0, b=0.0, epochs=2000):
    history = []
    for _ in range(epochs):
        g_w, g_b = backward_pass(x, y, w, b)
        w, b     = update_weights(w, b, g_w, g_b)
        history.append(compute_loss(y, forward_pass(x, w, b)))
    return w, b, history


if __name__ == "__main__":
    x = np.array([1., 2., 3., 4., 5., 6., 7., 8.])
    y = 3 * x + 5                 # the rule the model has to rediscover

    w, b, history = train(x, y)
    print(w, b)                    # 2.998259 5.007046
    print(history[-1])            # 1.65e-05

    assert abs(w - 3) < 0.05 and abs(b - 5) < 0.05
    assert history[-1] < history[0]

The run

What actually happened

Fast, then slow. And the two taps move at different speeds.

Most of the progress is immediate

389 to 8 in three epochs. Then 1997 more epochs to reach zero.

That shape comes straight from the geometry. Steep slope, big gradient, big step. Near the bottom the slope flattens, the gradient shrinks, the steps shrink with it. The method slows down as it arrives, for free.

The two taps do not have equal authority

Watch w. It hits 3.74 by epoch 10, overshooting its target of 3, then drifts back down for two thousand epochs. b is still at 0.84 at epoch 10 and does not reach 5 until the end.

The starting gradients say why: -198 for w, -37 for b. w gets multiplied by x, which runs from 1 to 8. A tap wired to large inputs has large influence and therefore a large gradient. A tap added at the end does not.

This is the first thing the shower warned about. One tap is coarse, one is fine, and a quarter turn does not mean the same on both.

They share one learning rate. The step that suits w starves b, and the run is bottlenecked by the slower tap.

Scaling up

From two taps to a wall of them

Swap the bathroom for a boiler room. A wall of taps, still one gauge

Every input feature connects to every output, so W becomes a grid. The five functions survive untouched. Multiplication becomes matrix multiplication.

Shapes are the contract. n examples, d_in features each, d_out numbers back. Inner dimensions cancel, outer ones survive. That is the only rule.

b is added to every row. NumPy handles that by broadcasting, so one row of biases serves a thousand examples.

One Change and this works for matrices as well

# shapes: X (n, d_in) | W (d_in, d_out) | b (1, d_out) | Y (n, d_out)

def forward_pass(X, W, b):
    return X @ W + b                       # b broadcasts down the rows