Files for this lesson

• src/conv_from_scratch.py — convolution by hand, output size, receptive field, shapes, and the parameter count of the tiny CNN.
• src/metrics.py — binary_report: accuracy and recall for each class.
• src/baselines.py — the extended file: the 1-NN from lesson 5, plus the most frequent class and the ink rules.
• src/compare_baselines.py — fits the baselines and prints the table.
• src/models/tiny_cnn.py and src/models/__init__.py — the network itself.
• src/train_tiny_cnn.py — training, with the checks and the final table.
• tests/test_conv_from_scratch.py, tests/test_baselines_simple.py, tests/test_metrics.py, tests/test_tiny_cnn.py — tests.

Reference if training or the tests disagree: lesson-07. You do not clone it. You add this lesson's files to your own repository and commit them.

Predict

"Predict" is not an exam. Write down what you expect before the tables and the training run.

Before you start, write the answers:

1. What accuracy does the naive model "always answer the most frequent class" get on val? And the simple rule "few bright pixels means empty": above 0.9, or below 0.8?
2. What receptive field (how many input pixels one cell of the feature map sees) does the last layer of the tiny CNN have? Is it comparable to the height of a digit (about 30 pixels)?
3. We apply cross-entropy twice to an untrained model on two classes. What number do you expect as the starting loss?

Three questions for this lesson

1. What is a convolution, really? We write it by hand in NumPy and match it to nn.Conv2d.
2. What do we compare the model to? "Accuracy 0.96" means nothing until you know what a foolish rule scores.
3. How do you tell that training works at all? Two checks before the full run: the starting loss, and overfitting a single batch.

Part 1. Convolution by hand

A convolution is a sliding window. Take a small patch of the picture, multiply it element by element by the kernel (a set of weights), add the products, and write down one number. Shift the window and repeat. The same kernel is applied everywhere, so a convolutional layer has few parameters and finds the same pattern wherever it sits.

An example worth checking on paper. A 3×3 picture with the numbers 1 through 9, and a 2×2 kernel:

input:    1 2 3         kernel: 1  0
          4 5 6                 0 -1
          7 8 9

The top-left window gives 1·1 + 2·0 + 4·0 + 5·(-1) = -4. The kernel is not flipped: nn.Conv2d computes cross-correlation. For a neural net that does not matter, because the weights are learned.

Output size: (n + 2·padding - k) // stride + 1 along each axis. With padding=1 and a 3×3 window the size stays the same. A 2×2 pool with stride 2 cuts it in half.

Receptive field. One output cell does not depend on the whole picture. It depends on a window of the input. After several layers that window grows.

No cell of the last map sees a whole digit (~30 px). The network builds its decision from local pieces.

Part 2. What to compare against: baselines

Labels: 1 = empty, 0 = value. The baselines:

• most frequent class: always the same answer;
• ink threshold: "little ink means empty" (two versions of the feature);
• 1-NN on pixels from lesson 5.

python src/compare_baselines.py --data-dir data/processed

Result on val, tag lesson-07. The script prints Russian labels; match the numbers:

модель                              accuracy  value_recall  empty_recall   (val)
всегда самый частый класс              0.622         1.000         0.000
порог по сумме пикселей                0.632         0.966         0.080
порог по числу ярких пикселей          0.709         0.789         0.578
1-NN по пикселям                       0.959         0.934         1.000

The first idea failed. The sum of pixels is about as good as the naive answer. After the inversion, the background of dull paper also "glows", so the sum measures how dull the paper is, not the ink.

The second idea is better, and still weak. The count of pixels brighter than 0.3 scores 0.709. The hard empty cells from lesson 4 break the rule:

Accuracy hides the losses. 1-NN has accuracy 0.959, but value_recall 0.934. That is why the table has three numbers, not one.

On test (another 38 sheets):

модель                              accuracy  value_recall  empty_recall   (test)
всегда самый частый класс              0.578         1.000         0.000
порог по сумме пикселей                0.592         0.986         0.053
порог по числу ярких пикселей          0.712         0.879         0.483
1-NN по пикселям                       0.983         0.970         1.000

The bar for the network: not worse than 0.99 on val. Linear models on pixels already land near that.

Part 3. The model

src/models/tiny_cnn.pyclass TinyCNN(nn.Module):
    def __init__(self, channels=(16, 32, 64), num_classes: int = 2) -> None:
        super().__init__()
        layers: list[nn.Module] = []
        c_in = 1
        for c in channels:
            layers += [
                nn.Conv2d(c_in, c, kernel_size=3, padding=1, bias=False),
                nn.BatchNorm2d(c),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(2),
            ]
            c_in = c
        self.features = nn.Sequential(*layers)
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.head = nn.Linear(c_in, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.features(x)
        x = self.pool(x).flatten(1)
        return self.head(x)

• Three blocks: convolution → BatchNorm → ReLU → MaxPool.
• bias=False on the convolution: BatchNorm supplies the shift.
• AdaptiveAvgPool2d(1) — the head does not care how wide the picture is.
• The head is a Linear layer onto two classes.

Shapes (computed by conv_from_scratch.py). The script prints Russian labels; match the numbers:

вход             (1, 48, 160)
conv1+BN+ReLU    (16, 48, 160)
pool1            (16, 24, 80)
conv2+BN+ReLU    (32, 24, 80)
pool2            (32, 12, 40)
conv3+BN+ReLU    (64, 12, 40)
pool3            (64, 6, 20)
global avg pool  (64,)
голова           (2,)
параметров: 23538

Part 4. Training and two checks

If either check fails, we stop. The bug is in the code or the data, not in "too few epochs".

Check 1: starting loss. On an untrained model with two classes, cross-entropy is about ln 2 ≈ 0.693.

Check 2: overfit one batch. In 200 steps the loss falls almost to zero, and accuracy on that batch is 1.000.

Both checks live inside train_tiny_cnn.py. You type the file yourself. If the starting loss or the batch overfit does not match, compare your file with the reference lesson-07 in the browser.

Commands

python src/conv_from_scratch.py
python src/compare_baselines.py --data-dir data/processed
python src/train_tiny_cnn.py --data-dir data/processed --overfit-batch
python src/train_tiny_cnn.py --data-dir data/processed --epochs 10 --out checkpoints/tiny_cnn.pt
python -m pytest tests/test_conv_from_scratch.py tests/test_baselines_simple.py tests/test_metrics.py tests/test_tiny_cnn.py -q

Short run

python src/conv_from_scratch.py
python src/compare_baselines.py --data-dir data/processed-smoke
python src/train_tiny_cnn.py --data-dir data/processed-smoke --overfit-batch
python src/train_tiny_cnn.py --data-dir data/processed-smoke --epochs 1 --max-batches 3 --out checkpoints/tiny_cnn_smoke.pt
python -m pytest tests/test_conv_from_scratch.py tests/test_baselines_simple.py tests/test_metrics.py tests/test_tiny_cnn.py -q

Expected

Output of conv_from_scratch.py (captured on tag lesson-07). The script prints Russian labels; match the numbers:

форма результата: (4, 12, 20)
naive vs fast: max |разница| = 2.86e-06
naive vs torch nn.Conv2d: max |разница| = 2.86e-06

recept. поле tiny CNN (слой: поле в пикселях входа, прыжок):
  conv1  поле   3 px, прыжок 1
  pool1  поле   4 px, прыжок 2
  conv2  поле   8 px, прыжок 2
  pool2  поле  10 px, прыжок 4
  conv3  поле  18 px, прыжок 4
  pool3  поле  22 px, прыжок 8
...
параметров: 23538

Overfitting one batch:

Device: mps
параметров: 23538
стартовый loss: 0.689 (ln 2 = 0.693)  OK
  шаг   0: loss 0.7339
  шаг  25: loss 0.0570
  ...
  шаг 200: loss 0.0015
overfit одного батча: loss 0.0015, accuracy 1.000  OK

Full training (10 epochs; the best one was epoch 7):

эпоха  1: train loss 0.1720 acc 0.946 | val loss 0.0941 acc 0.991 ...  <- лучшая
...
эпоха  7: train loss 0.0113 acc 0.999 | val loss 0.0119 acc 1.000 (value_recall 1.000, empty_recall 1.000)  <- лучшая
...
чекпоинт: checkpoints/tiny_cnn.pt (эпоха 7)

модель                              accuracy  value_recall  empty_recall   (val)
всегда самый частый класс              0.622         1.000         0.000
порог по сумме пикселей                0.632         0.966         0.080
порог по числу ярких пикселей          0.709         0.789         0.578
1-NN по пикселям                       0.959         0.934         1.000
tiny CNN (эпоха 7)                     1.000         1.000         1.000

Your device may be cpu or cuda, and the best epoch may be another one. What matters: starting loss near 0.69, the overfit check OK, val accuracy not worse than 0.99.

Tests: 20 passed (the whole suite of lessons 2–7: 90 passed).

Compare with the forecasts.

1. Most frequent class: 0.622 on val. The bright-pixel rule: 0.709 (below 0.8).
2. Receptive field of the last layer: 22 pixels, less than the height of a digit (~30).
3. Starting loss near ln 2 = 0.693 (ours was 0.689, OK).

Exercise

1. Remove BatchNorm. In TinyCNN, delete nn.BatchNorm2d(c) and give the convolution bias=True. How did the loss curve and the final accuracy change?
2. Halve the network. Set channels=(8, 16, 32). The parameter count becomes 6010 (tiny_cnn_summary(channels=(8, 16, 32))). How much accuracy did you lose?
3. Turn augmentation on. Add --augment and compare the train/val accuracy gap with the run that had none.

For each variant write down val accuracy, value_recall, and empty_recall.

If it fails

• ModuleNotFoundError: No module named 'torch'. The environment from lesson 1 is not active.
• Starting loss is far from 0.69. Check the labels (only 0 and 1), the input shape [B, 1, 48, 160], and the head Linear(64, 2).
• One batch will not overfit. lr too small or too large, swapped labels, the model left in eval, or "empty" data. Fix --overfit-batch first.
• val acc stuck at 0.62. The model always answers one class. Check model.train(), optimizer.step(), and lr.
• RuntimeError: mat1 and mat2 shapes cannot be multiplied. The head's input size is not 64.
• Training is slow on CPU. Use --max-batches for debugging, or --workers 2.
• Test test_matches_scipy_correlate was skipped. scipy is not installed; that is expected.

Checklist

• This lesson's files are in your repository and committed
• conv_from_scratch.py: naive ≈ torch, receptive field up to 22 px, 23538 parameters
• The baseline table on val has been read
• --overfit-batch → OK
• Full training: val acc ≥ 0.99, checkpoint saved
• pytest → 20 passed
• The three forecasts from "Predict" have been checked

FAQ

What's next

Argmax is not a production decision. The next lesson is Lesson 8. Decision threshold. The production recipe (class balance, early stopping, a config file) is lesson 9.

Reference for comparison: lesson-07. You do not clone it. Previous: Lesson 6. Dataset and augmentations.