An introduction to linear algebra
for neural networks in PyTorch

Why this matters

A neural network is made up of weights, biases and activation functions. All of which can be represented in the form of vectors. Vectorization helps solve complex computation faster to train neural networks. Let's look at step by step how they do so.

The dataset

House rent prediction

Let's consider this dummy dataset for house rent prediction. For each apartment we record two numbers: the floor area in hundreds of square feet, and the walking distance to the nearest metro station in kilometres. We also record the monthly rent.

One row of this table is a vector. The full table is a matrix. Predicting rent from a row uses the dot product. Changing the axes the data sits on uses a matrix transformation. We will take these one at a time.

import torch

# Each row is one flat: [area (100 sqft), metro distance (km)]
X = torch.tensor([[4.5, 1.2],
                  [6.0, 2.0],
                  [3.0, 0.5],
                  [8.0, 4.0],
                  [5.5, 3.2],
                  [7.0, 1.0]])      # torch.Size([6, 2])

y = torch.tensor([22., 27., 17., 31., 22., 34.])

w = torch.tensor([5.0, -2.0])
b = 2.0

Section 1

Scalars, vectors, matrices, and tensors

These four terms describe the same kind of object at different sizes.

The difference between them is how many indices you need to point at a single number inside. Click each one to see it in our dataset.

Section 2

What a vector is

A vector is an ordered list of numbers. Flat A is the vector [4.5, 1.2]. If we put area on the horizontal axis and metro distance on the vertical axis, that pair of numbers becomes a point, which we draw as an arrow starting at the origin.

Every vector has two properties: a length and a direction.

Direction is worth understanding separately from length. Flat A and a hypothetical flat [9.0, 2.4] point in exactly the same direction, because the second is just twice the first. They have the same ratio of area to distance. Think of it as two apartments with the same character, one simply larger than the other.

Drag the arrowhead in the chart below to see the length and direction update.

a = X[0]                          # tensor([4.5000, 1.2000])
torch.linalg.norm(a)              # tensor(4.6573)
a / torch.linalg.norm(a)          # tensor([0.9662, 0.2577])
torch.rad2deg(torch.atan2(a[1], a[0]))   # tensor(14.9314)

Section 3

Adding and subtracting vectors

To add two vectors, add the matching components.

On a chart this is the tip-to-tail rule. Walk along the first arrow, then start the second arrow from where you stopped. The point you end at is the sum.

Subtraction is the more useful operation in practice. D − C = [5.0, 3.5] tells us that flat D has 5 more units of area than flat C, and is 3.5 km further from the metro. That one vector describes the trade-off between the two listings.

Subtraction is also how we centre a dataset. Subtracting the average flat from every row moves the whole group of points so that its centre sits at the origin, which many algorithms expect.

X[0] + X[2]        # tensor([7.5000, 1.7000])
X[3] - X[2]        # tensor([5.0000, 3.5000])  — D compared to C

# Centring: dim=0 averages down the rows, broadcasting does the rest
Xc = X - X.mean(dim=0)

Section 4

Multiplying a vector by a number

Multiplying a vector by a single number, called a scalar, multiplies every component by that number.

The vector stays on the same line through the origin. Only its length changes. A negative scalar flips it to the opposite side of the origin. Think of it as a volume dial: the sound stays the same, only louder or softer, and a negative value inverts it.

This operation appears in two common places. The learning rate in gradient descent scales the update step: w ← w − η∇L, where the gradient supplies the direction and η supplies the size. Feature scaling divides each column by its standard deviation so that no feature dominates simply because of the units it was measured in.

2.0 * X[0]         # tensor([9.0000, 2.4000]) — twice as long
-1.0 * X[0]        # tensor([-4.5000, -1.2000]) — opposite

# correction=0 divides by n; torch divides by n-1 by default
Xs = (X - X.mean(0)) / X.std(0, correction=0)

Section 5

The dot product

The dot product takes two vectors and returns a single number. Multiply the matching components and add the results.

Suppose we believe rent rises by about ₹5,000 for each extra 100 square feet, and falls by about ₹2,000 for each kilometre from the metro, starting from a base of ₹2,000. Those beliefs form their own vector, w = [5, −2], called the weight vector. The dot product of the weights and a flat gives the predicted rent.

The second form explains what the number means. Because it contains the cosine of the angle between the two vectors, the dot product measures how much they point the same way. A positive result means they broadly agree, zero means they are at right angles and unrelated, and a negative result means they point in opposing directions. If you divide out both lengths, you are left with just the cosine, which is called cosine similarity and is used to compare users or documents.

The dot product also gives us projection: the shadow one vector casts on another when light shines perpendicular to it. Projection is the operation behind least-squares regression and principal component analysis.

Use the sliders below to rotate and lengthen x, and watch where the dot product changes sign.

w @ X[0] + b                  # tensor(22.1000)
preds = X @ w + b             # all six at once
# tensor([22.1000, 28.0000, 16.0000, 34.0000, 23.1000, 35.0000])

resid = preds - y
loss  = 0.5 * torch.mean(resid**2)       # tensor(1.1017)

# Cosine similarity has a built-in in PyTorch
cos = torch.nn.functional.cosine_similarity(X[0], X[5], dim=0)   # 0.993

proj = (w @ X[0]) / (w @ w) * w          # projection of x onto w

Section 6

Matrices as transformations

So far the matrix X has only stored data. A matrix can also do something. Given a 2×2 matrix M and any vector x, the product Mx is a new vector. Applying it to all six flats moves the entire dataset at once.

Those two properties are what make the transformation linear. In practice they mean three things you can see on the chart: straight lines stay straight, parallel lines stay parallel, and the origin does not move. A useful way to picture it is a sheet of graph paper that can be stretched, rotated, or slanted, but never crumpled or torn.

Reading a matrix from its columns

The first column of M is where the vector [1, 0] ends up. The second column is where [0, 1] ends up. Every other vector follows automatically, because every vector is a combination of those two. This is why matrix multiplication is defined the way it is. The two coloured arrows in the chart show these landing spots.

The determinant

The value ad − bc is the determinant. It tells you how much areas grow or shrink under the transformation. A determinant of 1 preserves area. A determinant of 0 means the transformation has flattened the plane onto a single line, and the original data cannot be recovered. A negative determinant means the plane has been flipped over.

Try the preset buttons below to see the four standard transformations.

t = torch.tensor(0.7854)
R = torch.tensor([[torch.cos(t), -torch.sin(t)],   # rotation
                  [torch.sin(t),  torch.cos(t)]])
S = torch.diag(torch.tensor([1.6, 0.6]))           # scaling
H = torch.tensor([[1.0, 0.8], [0.0, 1.0]])         # shear

Xr = X @ R.T          # .T works the same on a 2-D tensor
torch.linalg.det(R)   # tensor(1.) — rotation preserves area

# One transformation after another = multiplying the matrices
C = H @ R
torch.allclose(X @ C.T, (X @ R.T) @ H.T)   # True

Why this is useful here

In our dataset, area and metro distance are related: the larger flats tend to be further out. Principal component analysis finds the rotation matrix that lines the data up with the directions in which it actually spreads. After rotating, the first axis roughly measures overall size and quality, and the second measures the trade-off between them. Same six flats, more informative axes.

Xc = X - X.mean(0)
C  = Xc.T @ Xc / (len(X) - 1)        # 2x2 covariance matrix
vals, vecs = torch.linalg.eigh(C)    # tensor([0.7766, 4.2557])
Xpca = Xc @ vecs.flip(1)             # flip(1) reverses the columns

Section 7

Matrix operations in PyTorch

The sections above introduced each operation alongside the code that performs it. This section collects the core ones in one place, so you can see exactly what each returns.

Every block below shows the input on the left and the actual output of the notebook cell on the right. Run them in order — a, b, and x carry over between blocks.

Element-wise addition

Section 3, in code: the two vectors are combined component by component.

import torch

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

# Element-wise addition
element_add = a + b

print(element_add)

tensor([5, 7, 9])

Element-wise multiplication

The * operator is not matrix multiplication. It multiplies matching components, the same way + adds them.

# Element-wise multiplication
element_mul = a * b

print(element_mul)

tensor([ 4, 10, 18])

Dot product

Section 5, in code. torch.matmul() handles the dot product of two vectors, and the same function computes matrix–vector and matrix–matrix products — so X @ w from the rent model translates directly to torch.matmul(X, w). Check the result by hand: 1·4 + 2·5 + 3·6 = 32.

# Dot product
dot_product = torch.matmul(a, b)

print(dot_product)

tensor(32)

Broadcasting

When shapes differ but are compatible, PyTorch stretches the smaller tensor automatically. A row of shape [3] plus a column of shape [3, 1] produces the full [3, 3] table of pairwise sums — the same expansion the outer product performs with multiplication in the next section.

a = torch.tensor([1, 2, 3])
b = torch.tensor([[1],
                  [2],
                  [3]])

# Shapes [3] and [3,1] expand to [3,3]
c = a + b

print(c)
print(c.shape)

tensor([[2, 3, 4],
        [3, 4, 5],
        [4, 5, 6]])
torch.Size([3, 3])

Transpose

Rows become columns. This is the R.T from the transformation code in Section 6: our data stores vectors as rows, so we transpose the matrix before applying it.

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])

# Swap dimensions 0 and 1
transposed = x.transpose(0, 1)

print(transposed)
print(transposed.shape)

tensor([[1, 4],
        [2, 5],
        [3, 6]])
torch.Size([3, 2])

Reshape

The six numbers stay in order; only the shape changes. Shape mismatches are the most common error in PyTorch, and reshape() together with .shape is how you fix them.

# Same six numbers, new shape
reshaped = x.reshape(3, 2)

print(reshaped)
print(reshaped.shape)

tensor([[1, 2],
        [3, 4],
        [5, 6]])
torch.Size([3, 2])

Section 8

The outer product

The dot product turns two vectors into one number. The outer product turns two vectors into a matrix. Each entry is one component of the first vector multiplied by one component of the second, so the result is a table of every possible pairing.

It works like a multiplication table from school: the row headings run down one side, the column headings run across the top, and each cell holds the product of the two.

Building a covariance matrix

Take the outer product of each centred flat with itself, add all six results together, and divide by five. That gives the covariance matrix. The diagonal entries measure how much each feature varies on its own. The off-diagonal entry measures whether area and distance move together. In our data it is positive, which is why the two weights cannot be interpreted independently.

Updating a weight matrix

In a neural network layer, the gradient of the loss with respect to the weight matrix is the outer product of the output error with the input values. It says: adjust the connection from input i to output j in proportion to how active i was and how wrong j was.

Move the sliders and notice that the determinant stays at zero. Every row of an outer product is a multiple of the same vector, so the matrix always has rank 1.

torch.outer(X[0], X[0])
# tensor([[20.2500,  5.4000],
#         [ 5.4000,  1.4400]])

# Covariance built as a sum of outer products
Xc = X - X.mean(0)
C = sum(torch.outer(r, r) for r in Xc) / (len(X) - 1)
torch.allclose(C, torch.cov(X.T))        # True

delta = torch.tensor([0.1, -0.3])        # output error
grad_W = torch.outer(delta, X[0])        # torch.Size([2, 2])
torch.linalg.matrix_rank(grad_W)         # tensor(1)

Section 9

Dual vectors

Look at the weight vector w = [5, −2] again. It has two components and lives in the same plane as the flats, but it plays a different role. A flat is a thing we measure. w is the rule we measure it with: give it any flat, and it returns a rent.

A vector that works this way is called a dual vector. You may also see it called a covector, a linear functional, or a one-form. All four names mean the same thing.

A price list is a good analogy. A shopping basket is a vector of quantities. A price list is a dual vector: it takes a basket and returns a total. The two are written the same way, but you would never add a price list to a basket.

Drawing a dual vector

A dual vector is best drawn not as an arrow but as a set of evenly spaced parallel lines. Each line joins all the flats that receive the same rent. A flat sitting on the ₹25k line rents for ₹25k regardless of how that splits between area and distance. The value assigned to a vector is how many lines it crosses. Closely spaced lines mean a stronger rule, since a small step changes the answer a lot.

This is also the correct way to think about a gradient. The gradient answers the question "if I move the weights slightly in this direction, how much does the loss change?" It takes a direction and returns a number, which makes it a dual vector rather than an ordinary one.

# The dual vector written as a function
rent = lambda x: w @ x + b

rent(X[0])                          # tensor(22.1000)

# The gradient is a dual vector on the space of weights
resid = X @ w + b - y
grad_w = X.T @ resid / len(X)       # tensor([6.7500, 3.0233])
grad_b = resid.mean()               # tensor(0.8667)

w_new = w - 0.01 * grad_w           # scalar multiplication again

Section 10

Putting it all together

The code below fits the rent model. Every operation from the previous sections appears in it. Centring the data is vector subtraction. Standardising is scalar multiplication. Each prediction is a dot product. The gradient is a stack of dot products. The update step is scalar multiplication.

import torch

X = torch.tensor([[4.5,1.2],[6.0,2.0],[3.0,0.5],
                  [8.0,4.0],[5.5,3.2],[7.0,1.0]])
y = torch.tensor([22.,27.,17.,31.,22.,34.])

mu, sd = X.mean(0), X.std(0, correction=0)
Xs = (X - mu) / sd

w, b, eta = torch.zeros(2), torch.zeros(1), 0.1

for step in range(600):
    pred  = Xs @ w + b              # six dot products
    resid = pred - y
    loss  = 0.5 * torch.mean(resid**2)

    gw = Xs.T @ resid / len(Xs)     # the gradient
    gb = resid.mean()

    w -= eta * gw                   # scalar multiplication
    b -= eta * gb

print(w, b, loss)
# tensor([ 7.3401, -3.0207]) tensor([25.5000]) tensor(0.0077)

# The same answer without a loop, using a projection
A = torch.cat([Xs, torch.ones(len(Xs), 1)], dim=1)
theta = torch.linalg.pinv(A) @ y    # tensor([7.3401, -3.0207, 25.5000])

# The shape of the problem, from the covariance matrix
C = torch.cov(Xs.T)
torch.linalg.eigvalsh(C)            # tensor([0.4035, 1.9965])

# PyTorch can also do the calculus for you
w = torch.zeros(2, requires_grad=True)
loss = 0.5 * torch.mean((Xs @ w - y)**2)
loss.backward()
w.grad                              # the same gw, computed automatically

The last two lines connect back to Section 6. The eigenvalues of the covariance matrix tell you how easy this problem is for gradient descent. If one eigenvalue is much larger than the other, the loss surface is a long narrow valley, and the optimiser bounces between the walls on its way down. Rotating the data into the eigenvector directions turns that valley into a round bowl, and the optimiser walks straight to the bottom. That rotation is a matrix transformation, which is why the geometry and the training loop are two views of the same thing.