Tensor Shapes, Types and Broadcasting
in PyTorch

Overview

What you will learn

A tour of the tensor operations that cause the most errors, and how to stop them.

Tensors are the data structure PyTorch uses for everything. The data you load is a tensor. The input to a model is a tensor. The output is a tensor. The weights inside the model are tensors.

This means that when something breaks in PyTorch, it usually breaks at a tensor. A wrong shape. A wrong data type. A missing batch dimension.

In this tutorial, you will discover how to work with tensor shapes, data types and broadcasting in PyTorch.

After completing this tutorial, you will know:

• How to read a tensor shape, and why the first number behaves differently from the rest.
• How PyTorch chooses a data type for you, and how to take control of it.
• How to add and remove dimensions so that a model will accept your data.
• How broadcasting removes loops from your code, and the two rules that govern it.
• Five behaviors that fail quietly and will cost you debugging time.

Let's get started.

Tutorial Overview

This tutorial is divided into eleven parts; they are:

1. Setup and Prerequisites
2. How to Read a Tensor Shape
3. How to Control Tensor Data Types
4. Four Ways to Create a Tensor
5. How to Add and Remove Dimensions
6. How to Index and Slice a Tensor
7. How to Do Element-Wise Math
8. How Broadcasting Works
9. More Tensor Operations
10. Five Surprising Behaviors
11. A Checklist for Debugging Tensor Errors

1 — Setup and Prerequisites

Setup and Prerequisites

What you need before you start, and where the material comes from.

This tutorial assumes you are comfortable with Python lists and slicing, and that you have seen NumPy arrays before — enough to know what a shape is. You do not need to have trained a model.

Three terms are used throughout without being explained again:

• A layer is one step of a model. nn.Linear(in, out) is the simplest one. It multiplies its input by a matrix of weights and adds a bias.
• A sample is one row of your data.
• A batch is a stack of samples handed to the model together.

All of the examples below assume the following four imports.

import torch
import torch.nn as nn
import numpy as np
import pandas as pd

Each section is self-contained. Where an example depends on a variable created earlier, that variable is created again so you can run any section on its own.

2 — Tensor Shapes

How to Read a Tensor Shape

The shape is the first thing to check when something goes wrong.

You have probably been using tensors already without thinking about them. That works until it doesn't.

The example below creates a tensor holding six delivery distances and prints its shape.

import torch

# Your delivery data
distances = torch.tensor([[3.0], [7.0], [12.0], [18.0], [22.0], [28.0]])
print(distances.shape)

Running the example prints the shape of the tensor.

torch.Size([6, 1])

You can see that the shape has two numbers, and they mean different things.

The first number, 6, is the batch size. There are six samples. The second number, 1, is the number of features per sample. Each sample holds one distance.

Anatomy of a shape. The first dimension is how many; the rest describe what each sample looks like.

A useful way to think about this is a stack of papers. The model reads each page the same way, whether there are six pages in the stack or 600.

This is why the batch size rarely causes trouble inside a layer. nn.Linear checks the number of features and does not care how many rows you handed it. Within a layer, the feature count is the thing that has to match.

A Shape That Works and a Shape That Fails

The example below builds a model that expects one feature per sample, passes it data with one feature, then passes it data with three.

import torch
import torch.nn as nn

distances = torch.tensor([[3.0], [7.0], [12.0], [18.0], [22.0], [28.0]])

simple_model = nn.Linear(1, 1)      # expects 1 feature per sample
output = simple_model(distances)    # shape [6, 1] — works

# Now with 3 features: distance, hour, weather
features = torch.tensor([[3.0, 7.0, 1.0],
                         [18.0, 22.0, 2.0]])   # shape [2, 3]
output = simple_model(features)

Running the example results in an error, as follows.

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x3 and 1x1)

The model was built for one input feature. It got three.

3 — Data Types

How to Control Tensor Data Types

PyTorch picks a data type from what you type. Learn the rule and you control it.

There are two cases that cover almost everything you will write:

• Whole numbers give int64.
• Numbers with a decimal point give float32.

For completeness, booleans give torch.bool and complex literals give complex64, but neither comes up often when you are starting out.

The example below creates one tensor of each kind and prints the data type.

import torch

int_tensor = torch.tensor([1, 2, 3])
print(int_tensor.dtype)

float_tensor = torch.tensor([1.0, 2.0, 3.0])
print(float_tensor.dtype)

Running the example prints the two data types.

torch.int64
torch.float32

Relying on a decimal point is fragile. You can be explicit instead, using either of the two routes below. Both guarantee float32 even if you forget the decimal point.

float_tensor = torch.tensor([1, 2, 3], dtype=torch.float32)
float_tensor = int_tensor.float()   # convert any tensor to float32

Mixing Data Types No Longer Fails

PyTorch used to raise an error when you combined tensors of different data types. Since version 1.5 it resolves them through type promotion: int meets float, int becomes float.

The rules are close to Python's but not identical. Part 10 covers the case where they come apart.

float_tensor = torch.tensor([1.0, 2.0, 3.0])
int_tensor   = torch.tensor([1, 2, 3])

mixed_tensor = float_tensor + int_tensor
print(f"mixed type: {mixed_tensor.dtype}")

Running the example prints the promoted data type.

mixed type: torch.float32

Which Data Type Should You Use?

4 — Creating Tensors

Four Ways to Create a Tensor

Lists, NumPy, DataFrames and built-in patterns. Each one has a catch.

Let's take a closer look at each in turn.

1. From a Python List

This is the simplest way, and the one you will use for small examples.

import torch

x = torch.tensor([1, 2, 3], dtype=torch.float32)
x, x.shape

Running the example prints the tensor and its shape.

(tensor([1., 2., 3.]), torch.Size([3]))

2. From a NumPy Array

PyTorch tensors behave almost exactly like NumPy arrays, and conversion is a single call.

import torch
import numpy as np

numpy_array = np.array([[1, 2, 3], [4, 5, 6]])
x = torch.from_numpy(numpy_array)
x, x.shape

Running the example prints the converted tensor.

(tensor([[1, 2, 3],
         [4, 5, 6]]),
 torch.Size([2, 3]))

3. From a pandas DataFrame

There is no direct function for this. You extract .values and convert that.

import torch
import pandas as pd

df = pd.DataFrame({
    'distance_miles': [1.6, 13.09, 6.97],
    'delivery_times': [7.22, 32.41, 17.47]
})
torch.tensor(df.values)

Running the example prints the tensor.

tensor([[ 1.6000,  7.2200],
        [13.0900, 32.4100],
        [ 6.9700, 17.4700]], dtype=torch.float64)

Note the dtype=torch.float64 at the end of that output. NumPy's default data type has leaked through the DataFrame. A single call to .float() fixes it.

4. From a Built-In Pattern

These give you test data on demand without typing numbers.

torch.zeros(3, 2)
torch.ones(3, 2)
torch.rand(3, 2)          # uniform between 0 and 1
torch.arange(0, 10, 1)

Running the example prints the four tensors.

tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])
tensor([[1., 1.],
        [1., 1.],
        [1., 1.]])
tensor([[0.3003, 0.1609],
        [0.0812, 0.9012],
        [0.7915, 0.3273]])
tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

5 — Reshaping

How to Add and Remove Dimensions

Models expect a batch dimension. Forgetting it is the shape error I hit most.

Say you want a single prediction for a delivery of 25 miles. That is one number, a scalar. Your model expects [batch_size, features], which for one sample with one feature means [1, 1].

The example below shows what you actually have.

import torch

single_distance = torch.tensor(25.0)
print(single_distance.shape)

Running the example prints an empty shape.

torch.Size([])

No dimensions at all. The unsqueeze() function adds them, one index at a time.

with_batch      = single_distance.unsqueeze(0)   # torch.Size([1])
ready_for_model = with_batch.unsqueeze(1)        # torch.Size([1, 1]) — ready

Watch the brackets grow. Each call to unsqueeze wraps the tensor in one more pair.

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])          # [2, 3]

expanded = x.unsqueeze(0)
expanded.shape, expanded

Running the example prints the new shape and the extra bracket.

(torch.Size([1, 2, 3]),
 tensor([[[1, 2, 3],
          [4, 5, 6]]]))

Going the other way, squeeze() removes dimensions of size one. This is useful for cleaning up after a batch has been through a model.

expanded2 = expanded.unsqueeze(0)     # [1, 1, 2, 3]
sq = expanded2.squeeze()
sq.shape, sq

Running the example prints the squeezed tensor.

(torch.Size([2, 3]),
 tensor([[1, 2, 3],
         [4, 5, 6]]))

Note the difference between the two forms. Calling squeeze(1) removes only the size-1 dimension at index 1, while a bare squeeze() removes all of them.

6 — Indexing & Slicing

How to Index and Slice a Tensor

Checking predictions and grabbing samples works just like Python lists.

The example below creates a small tensor and pulls four different pieces out of it.

import torch

x = torch.tensor([
    [1,  2,  3,  4],
    [5,  6,  7,  8],
    [9, 10, 11, 12]
])                                    # [3, 4]

x[0]        # first row
x[:2]       # first two rows
x[:, 0]     # first column of all rows
x[:, ::2]   # every other column

Running the example prints the four selections.

tensor([1, 2, 3, 4])
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])
tensor([1, 5, 9])
tensor([[ 1,  3],
        [ 5,  7],
        [ 9, 11]])

The pattern is [start:end:step], and the end index is not included.

Converting a Tensor to a Python Value

A single indexed value is still a tensor. The .item() function converts it to a plain Python number, but only for tensors holding exactly one element. Call it on anything larger and it fails.

x.item()          # 12 elements — fails
x[0, -1].item()   # 1 element — works

Running the example raises an error on the first line and prints a number on the second.

RuntimeError: a Tensor with 12 elements cannot be converted to Scalar
4

Selecting a Feature Across Every Sample

With more than one feature, you index across both dimensions. Column 0 pulls every distance.

data = torch.tensor([[ 3.0,  8.0, 1.0],   # distance, hour, weather
                     [ 7.0, 17.0, 2.0],
                     [12.0, 12.0, 1.0]])

distances = data[:, 0]

Running the example prints one value per sample.

tensor([ 3.,  7., 12.])

7 — Tensor Math

How to Do Element-Wise Math

One expression covers every sample at once, with no loop.

The smallest thing a neural network does is take a number, scale it by a weight, and add a bias.

Predicting delivery time from distance might use 2.3 minutes per mile plus 8 minutes of fixed overhead. The example below applies that to three deliveries in one line.

import torch

distances = torch.tensor([[3.0], [7.0], [12.0]])
weight, bias = 2.3, 8.0

weight * distances + bias
# [[2.3*3.0  + 8.0],
#  [2.3*7.0  + 8.0],
#  [2.3*12.0 + 8.0]]

Tensor math in PyTorch is element-wise. Each element is operated on independently. The same weight applies to every distance and the same bias is added to every result.

The code reads like ordinary Python, but it runs on all elements at once.

This works for a scalar, as above, and for two tensors of the same shape.

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

a + b
a * b
a / b

Running the example prints the three results.

tensor([5, 7, 9])
tensor([ 4, 10, 18])
tensor([0.2500, 0.4000, 0.5000])

Note the last line. Dividing two integer tensors gives you a float tensor. Part 10 returns to this.

8 — Broadcasting

How Broadcasting Works

Specify a value once and PyTorch applies it everywhere it is needed.

Consider three deliveries with three features each. You want to apply adjustment factors: 1.1× for distance, no change for time, and a 5× penalty for bad weather.

Repeating the row three times works, and it is redundant.

import torch

data = torch.tensor([[ 3.0,  8.0, 1.0],   # distance, hour, weather
                     [ 7.0, 17.0, 2.0],
                     [12.0, 12.0, 1.0]])  # shape [3, 3]

data * torch.tensor([[1.1, 1.0, 5.0],
                     [1.1, 1.0, 5.0],
                     [1.1, 1.0, 5.0]])   # same shape, three copies

Broadcasting lets you specify the values once instead.

data * torch.tensor([[1.1, 1.0, 5.0]])  # one row does it all

Both produce the same result. No loops, no manual repetition.

You have already seen this, in fact. It is how a scalar weight and bias applied to every distance at once in the previous section. A scalar is the smallest possible broadcast.

The Two Rules

Broadcasting follows two rules, applied in this order:

1. If the two shapes have a different number of dimensions, pad the shorter one with 1s on the left until they match.
2. For each dimension, if one size is 1, expand it to match the other. If neither is 1 and they differ, the operation fails.

Take a (1, 3) tensor combined with a (3, 1) tensor. Rule 1 does nothing, since both already have two dimensions. Rule 2 fires twice: 1 against 3 expands the 1, then 3 against 1 expands the 1. Both become (3, 3).

(1, 3) + (3, 1) → (3, 3). Dashed cells are the copies PyTorch makes for you. Each dimension of size 1 stretches to match the other tensor.

When the Shapes Have Different Lengths

Rule 1 is the one that gets forgotten, because it is invisible in the code.

What happens with [3] and [3, 1] — one dimension against two? PyTorch lines the shapes up from the right and pads the shorter one with 1s on the left. So [3] is read as [1, 3], and you are back to the case above.

Which is why the example below works, even though the two shapes look incompatible.

a = torch.tensor([1, 2, 3])       # shape [3]
b = torch.tensor([[1],
                  [2],
                  [3]])           # shape [3, 1]

a + b

Running the example prints a 3×3 tensor.

tensor([[2, 3, 4],
        [3, 4, 5],
        [4, 5, 6]])

9 — More Operations

More Tensor Operations

A short reference for the operations you will reach for next.

Skim this section now and come back to it when you need one of them.

Reshape and Transpose

import torch

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])         # [2, 3]

x.reshape(3, 2)
x.transpose(0, 1)

Running the example prints two tensors with the same shape.

tensor([[1, 2],
        [3, 4],
        [5, 6]])
tensor([[1, 4],
        [2, 5],
        [3, 6]])

The same numbers, arranged differently — and these are not the same operation.

reshape reads the elements in order and refills the new shape row by row. transpose swaps two dimensions, so element [i, j] becomes [j, i]. Same output shape here, different contents.

One thing worth knowing now and understanding later: reshape gives you a view onto the same memory when it can, and quietly makes a copy when it cannot. Part 10 shows you how to tell which one you got.

Combining Tensors with torch.cat

x1 = 20 * x

torch.cat((x, x1), dim=0)   # stack rows
torch.cat((x, x1), dim=1)   # stack columns

Running the example prints the two combined tensors.

tensor([[  1,   2,   3],
        [  4,   5,   6],
        [ 20,  40,  60],
        [ 80, 100, 120]])          # [4, 3]
tensor([[  1,   2,   3,  20,  40,  60],
        [  4,   5,   6,  80, 100, 120]])   # [2, 6]

All tensors must match in every dimension except the one being concatenated. Get it wrong and the error names the culprit for you.

RuntimeError: Sizes of tensors must match except in dimension 0.
Expected size 3 but got size 4 for tensor number 1 in the list.

Boolean Masks and Dot Products

x = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
x[x > 3]                    # boolean masking

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])
torch.matmul(a, b)          # also written a @ b

Running the example prints the masked values and the dot product.

tensor([ 4,  5,  6,  7,  8,  9, 10, 11, 12])
tensor(32)

Putting the Moves Together

Real feature engineering chains these operations. The example below slices out two columns, builds a boolean mask from them, converts it, and attaches it as a new feature.

# columns: [distance_km, hour_of_day]
trip_data = torch.tensor([[ 4.2,  9.0],
                          [15.7,  8.5],
                          [12.1, 18.0],
                          [ 3.3, 13.0]])   # shape [4, 2]

distances = trip_data[:, 0]
hours     = trip_data[:, 1]

is_long_trip    = distances > 10.0
is_morning_rush = (hours >= 8.0) & (hours < 10.0)
is_evening_rush = (hours >= 17.0) & (hours < 19.0)

mask = (is_morning_rush | is_evening_rush) & is_long_trip

new_feature_col = mask.float().unsqueeze(1)      # 1D bool → [N, 1] float
enhanced = torch.cat((trip_data, new_feature_col), dim=1)

Running the example gives a [4, 3] tensor, with the two long rush-hour trips flagged.

Slice out the columns you need. Build masks with >, & and |. Cast, unsqueeze and concatenate. That is four sections of this tutorial in eight lines of code.

10 — Surprising Behaviors

Five Surprising Behaviors

None of these are in the first page of the docs. All five cost me time.

The five behaviors below were found by running experiments rather than by reading. Four of them fail quietly, which is what makes them expensive.

1. A Python Float Will Not Give You float64

A Python float is 64-bit, so multiplying an integer tensor by 2.0 ought to give you float64. It does not.

import torch

i64 = torch.tensor([1, 2])                       # int64

(i64 * 2.0).dtype                                # a plain Python float
(i64 * torch.tensor([2.0], dtype=torch.float64)).dtype   # a float64 tensor

Running the example prints two different data types.

torch.float32
torch.float64

PyTorch treats bare Python numbers as weak. They take part in the arithmetic but do not get a vote on the result's data type, so you land on the default, float32. A float64 tensor is strong, and does promote.

This is the place where PyTorch's promotion rules come apart from Python's. It is also why a stray float64 in your pipeline almost always traces back to NumPy or pandas rather than to a number you typed.

2. squeeze() on the Wrong Dimension Fails Silently

Calling squeeze(1) removes dimension 1 if its size is 1. If it is not, you might expect an error. You get your tensor back unchanged.

t = torch.zeros(2, 3)
t.squeeze(1).shape       # dimension 1 has size 3, not 1

Running the example prints the original shape.

torch.Size([2, 3])

No warning and no exception. The bug shows up three lines later as a shape mismatch that points at entirely the wrong place. Print the shape immediately after any squeeze you were not completely sure about.

3. reshape Sometimes Copies, and view Refuses To

reshape returns a view of the same memory when the layout allows it, and copies when it does not. The data_ptr() function tells you which one you got.

x  = torch.tensor([[1, 2, 3],
                   [4, 5, 6]])

x.reshape(3, 2).data_ptr() == x.data_ptr()     # fresh, contiguous tensor

xt = x.transpose(0, 1)                         # transposing scrambles the layout
xt.is_contiguous()
xt.reshape(2, 3).data_ptr() == xt.data_ptr()

Running the example prints three booleans.

True
False
False

Now ask view to do the same job on the transposed tensor. It declines outright rather than copying behind your back.

xt.view(2, 3)

Running the example results in an error, as follows.

RuntimeError: view size is not compatible with input tensor's size and stride
(at least one dimension spans across two contiguous subspaces).
Use .reshape(...) instead.

That is the whole difference between the two functions. view is a promise of no copy, and it errors when it cannot keep that promise. reshape makes no promise and always succeeds.

Reach for reshape unless you specifically need the guarantee.

4. Two transpose Calls Are the Same Call

torch.equal(x.transpose(0, 1), x.transpose(1, 0))   # argument order
torch.equal(x.transpose(0, 0), x)                   # same dimension twice

Running the example prints True twice.

True
True

transpose swaps a pair of dimensions, and a pair has no order. This means (0, 1) and (1, 0) are one operation written two ways.

Passing the same index twice asks it to swap a dimension with itself, which does nothing. Neither case raises an error. Both are silent no-ops if you meant something else.

5. Integer Division Gives You Floats

a = torch.tensor([1, 2, 3])     # int64
b = torch.tensor([4, 5, 6])     # int64

(a / b), (a / b).dtype

Running the example prints a float tensor.

(tensor([0.2500, 0.4000, 0.5000]), torch.float32)

The / operator is true division, exactly as in Python 3, so two integer tensors give you a float tensor. Use // if you wanted the floor.

This is worth knowing because it is a place where a float32 can appear in a pipeline you thought was integers from end to end.

11 — Debugging Checklist

A Checklist for Debugging Tensor Errors

Six checks, in order. Most errors fall to the first two.

When a tensor error hits, work down this list rather than guessing.

1. Print tensor.shape. Knowing the current dimensions is the first step to any fix.
2. Read the error message. It names both shapes. Seen side by side, the fix is usually obvious.
3. Check that the batch dimension exists. A single sample needs [1, 1] at minimum.
4. Check the data type. Use float32 for neural networks, and cast with .float() or dtype=.
5. Check the feature count against what the model was built for.
6. If two tensors are being compared — a loss against targets, a mask against data — check that every dimension lines up, batch included.

Further Reading

Further Reading

Resources if you want to go deeper on any part of this tutorial.

APIs

• torch.Tensor — the full list of tensor methods.
• Broadcasting semantics — the formal version of Part 8.
• Tensor attributes — data types and the type promotion rules.
• torch.reshape and torch.Tensor.view — the copy-versus-view distinction from Part 10.

Courses

• PyTorch Fundamentals on DeepLearning.AI, taught by Laurence Moroney.

Summary

Summary

What you covered, and where to go next.

In this tutorial, you discovered how to work with tensor shapes, data types and broadcasting in PyTorch.

Specifically, you learned:

• That the first number in a shape is the batch size and the rest describe one sample, and that the batch size only stops mattering inside a layer.
• That PyTorch gives you int64 for whole numbers and float32 for decimals, and how to override both.
• How unsqueeze and squeeze add and remove the batch dimension that models require.
• The two rules of broadcasting — pad the shorter shape on the left, then expand any dimension of size 1 — and why the first one is easy to miss.
• Five behaviors that fail silently, including weak scalar typing and the hidden copy inside reshape.

The next thing to reach for is the training loop, where these shapes start moving through a model and back again.