How a Machine Learns to See a Face — and Read a Feeling

Where I got stuck, and how I got unstuck

When I first saw a neural-network formula full of Greek letters, I could read it out loud but I did not really understand it. I knew the words — weights, activation, convolution — without having the picture in my head. So I sat down and rebuilt the idea from the bottom, one small step at a time. This post is that rebuild, written the way I would explain it to a friend.

I keep one simple goal the whole way: get a machine to look at an image and answer one question, "is there a face here?" That single question covers almost every core idea in a CNN. In Part 2, I take the same idea and turn it into a live emotion detector.

The machine starts with numbers, so that is where I start too.

An image is just a grid of numbers

A computer never sees a "face" or a "3." It sees a grid of dots, and each dot — a pixel — is a single number describing how bright that dot is.

The scale runs from 0 (black) to 1 (white), with greys in between. So a 28×28 image is just 784 brightness numbers stacked in a square.

One more detail: the images here are grayscale — one number per pixel. A color photo uses three numbers per pixel (red, green, blue). Same idea, just stacked three deep.

That array — nothing more — is everything the network ever receives.

A neuron is just a weighted vote

Before CNNs, meet the smallest unit: the neuron. Here is the single-layer network equation that used to scare me:

Read it inside-out. There are only two steps.

Step 1 — each hidden neuron votes. Take every input Xⱼ (a pixel), multiply it by a weight wₖⱼ, add them all up, then add a baseline number called the bias wₖ₀. Squash the result with an activation g (it bends the math so the network can learn curves, not just straight lines). That gives one neuron's output.

Step 2 — the output neuron votes on the votes. Take all the hidden-neuron outputs, weight each by βₖ, add them up, add a bias β₀. That final number is the answer.

• w = weights from input → hidden
• β = weights from hidden → output
• the …₀ terms = biases (a default nudge)
• g = the activation (the squashing function)

Why not just feed every pixel to a plain network?

You could — but it falls apart fast. Here is what a CNN fixes.

1. It uses far fewer weights

In a plain network, every pixel connects to every hidden neuron. 784 pixels × 100 neurons = 78,400 weights — for a tiny image. A real photo (1000×1000 = a million pixels) explodes. A CNN instead uses one small filter (say 3×3 = nine numbers) and slides those same nine numbers across the whole image. Reuse beats uniqueness. This is called parameter sharing.

2. It tolerates small shifts

Because the same filter slides over every position, an eye in the top-left or nudged a few pixels right triggers the same detector wherever it lands. The network isn't glued to one spot.

3. It uses local correlations

Nearby pixels belong together — an edge, an eye, a curve is a local cluster. A 3×3 filter looks at a neighborhood all at once, so it captures these local patterns. Stack filters and you climb a ladder: edges → parts (an eye, a nose) → whole faces.

Filters and convolution

A filter is a tiny grid of numbers that acts as a pattern detector. The one we'll use looks for "a dark center surrounded by bright pixels" — exactly like an eye on skin:

Three questions everyone has, answered:

• Why do the numbers look random? At the start of training they are random. Training slowly tunes them until they become useful detectors. This filter is hand-picked so you can see the result — real CNNs learn their filters.
• Why many filters, not one? One filter spots one pattern. A face needs many — vertical edges, curves, eye-blobs — so you run a whole bank of filters, each producing its own map.
• Why slide it around? Because the feature could be anywhere. Sliding checks every location with the same detector.

Convolution is that sliding: lay the filter on a 3×3 patch, multiply matching cells, add the nine results into one number. A high number means "the pattern is here." Do it everywhere and you get a new grid called the feature map.

The same 3×3 filter sliding across the image — one number computed at every stop.

Let's do one stop by hand

Take the top-left patch (it sits over the left eye) and line up the filter:

Multiply each pair, then add the nine results. The eight outer cells use the +1 weight; the dark center (the eye) uses −8:

A big positive number means the filter found its "dark center, bright surround" pattern. That single 6.0 becomes one cell of the feature map.

Throw away the weak signals

After convolution, some numbers are negative — meaning "this patch does not match." We only care where the pattern is present, so we zero out everything negative. That's ReLU:

In the playground above, hit Apply ReLU and watch the negative cells snap to 0. The strong eye-signals stay; the noise disappears. This one trick is what lets the next step focus only on real evidence.

From features to a "yes, that's a face"

Now we turn the feature map into a decision. A simple classifier takes a few key values, weights them, and adds a bias:

The −2.5 bias keeps the model cautious so it won't shout "face!" at everything. Then the sigmoid squashes that score into a probability between 0 and 1:

Rule of thumb: score above 0 → probability above 0.5 → "face." Below 0 → "not a face." Play with it:

That completes the core CNN: image → filter (convolution) → ReLU → classifier → sigmoid → decision. If you can narrate that one line and explain each arrow, you understand the engine.

End to end, from pixels to a yes or no

Here is the complete path one image takes through the network. Every step is something we built above — now stacked in order.

Step through it, click the stages, or hit Play. Our toy version is the bare-minimum chain — real networks repeat the convolution-and-pooling block many times before the final decision.

Part 2 extends step 5 from one sigmoid to softmax (seven emotions at once) and wraps the whole flow in a sliding-box search so it works on a full photo, not just a single patch.

The rest of the CNN toolkit

What we built is the skeleton. Real CNNs add a few more parts on top. Here are the big ones, kept plain — this is roughly the level the Stanford CS-230 cheatsheet covers, just unpacked.

Stride and padding

Stride (S) is how far the filter jumps each step. Stride 1 moves one pixel at a time (lots of overlap); stride 2 skips every other spot, giving a smaller output.

Padding (P) adds a border of zeros around the image so the filter can reach the edges. Three common modes: valid (no padding — the output shrinks, which is why our 5×5 became 3×3), same (just enough padding so the output stays the same size as the input), and full (the most padding, so the filter sees every edge end to end).

The one formula that ties it together

Given input size I, filter size F, padding P, and stride S, the output size is:

Check it on our toy face: I = 5, F = 3, P = 0, S = 1 → (5 − 3 + 0) / 1 + 1 = 3. That is exactly why our feature map came out 3×3.

Pooling — shrink and toughen

After convolution, a pooling layer downsamples the feature map: slide a small window and keep one number per window. Max pooling (keep the largest value) is the most common and keeps the strongest signal; average pooling keeps the mean. Pooling shrinks the data and adds a bit more tolerance to small shifts. It has no weights to learn.

Many filters = depth

One filter finds one pattern and makes one feature map. Use K filters and you get K stacked feature maps — that stack is the layer's depth. On a color image with C channels, each filter is actually F × F × C, and K filters produce an output of size O × O × K.

The real architecture

Production CNNs repeat the convolution-and-pooling block several times — each round shrinks the spatial size and grows the depth — then flatten into fully-connected layers and finish with softmax. The interactive pipeline above walks the simplest version of that chain end to end.

Softmax, the sliding-box search, IoU, and non-maximum suppression are the next layer up — and they're exactly what Part 2 uses to go from one face to seven live emotions. Full reference: the Stanford CS-230 CNN cheatsheet by Afshine and Shervine Amidi.

Follow one image through all six levels

Reading about layers is one thing; watching the numbers move is another. Pick one of the tiny 8×8 images in the panel below and step it through every level — pixels, convolution, ReLU, pooling, flatten, and the final decision. It is a toy with one hand-picked filter, built so you can see exactly what happens at each stage.

One face, eight possible feelings

We ended the first half on a single sigmoid: one number that answered one yes-or-no question, "is this a face?" Emotions need a different ending. Now there are eight possible answers — neutral, happiness, surprise, sadness, anger, disgust, fear, contempt — and exactly one of them should win.

The tool for that is softmax. It takes the eight raw scores the network produces and turns them into eight probabilities that always add up to 100%. A sigmoid squashes one number on its own; softmax makes the options compete. Raise the score for one emotion and the others have to give way, because the total is fixed. The biggest score becomes the prediction, but the runners-up still get a share — which matters, because real faces are rarely one clean feeling.

About 35,000 faces — wildly uneven ones

I trained on FER+, a cleaned-up version of the well-known FER-2013 set: roughly 35,000 small grayscale faces, each 48×48 pixels. What makes FER+ good is how it was labeled — ten people voted on each face across the eight emotions, instead of one person assigning one tag.

The catch is the balance. The classes are lopsided in the extreme: about 12,900 neutral faces, but only 216 labeled contempt. Trained naively, a model just learns to shout "neutral" and "happy" and ignore the rare ones, because that alone scores well.

I used two fixes. The first is obvious: weight the rare emotions more heavily in the loss so the model can't afford to ignore them. The second is the one that actually mattered, and it comes from how FER+ was built.

A plain, classical CNN

Nothing exotic here — it's the same family of network from Part 1. Three convolution blocks (each one: convolution, then a normalization step, then ReLU, then pooling) shrink the 48×48 image down to 6×6 while the depth grows from 1 channel to 64 to 128 to 256. Then two fully-connected layers turn all of that into the eight emotion scores, and softmax finishes the job.

That's about 2.7 million parameters — small by modern standards — trained from scratch on a single GPU (an NVIDIA DGX Spark). No pretrained weights, no transformer, no tricks beyond the two above. I wanted to see how far an honest, textbook CNN gets on its own.

Hard labels, then the vote distribution

I trained the exact same network twice, changing only the targets.

Round one — hard labels. Each face got its single winning emotion. This reached 78.1% on the held-out test set. Respectable, but it overfit: by around epoch 60 it had memorized the training set (95% accuracy there) while the test number had stopped moving. It was also overconfident, often very sure even when wrong.

Round two — soft labels. Same architecture, same optimizer, same number of epochs. The only change was feeding it the vote distribution from Section 02 instead of the single winner. Test accuracy rose to 80.1% — a clean two points — and, just as useful, the overconfidence went away. Now when the model is right it's sure, and when it's wrong it hesitates around 50%. That honesty is what makes the live demo below feel sensible rather than erratic.

The honest scorecard

Overall test accuracy is 80.1%. But the average hides the real story, which is that the model is excellent at the common emotions and shaky at the rare ones. Here is the per-class breakdown on faces it never saw during training.

Per-class results on the held-out test set (v2, soft labels).

The pattern is clear. Happiness, neutral, and surprise are distinct and well-represented, so the model nails them. Disgust, fear, and contempt are rare — contempt has only 216 examples in the entire dataset — so the model barely learns them. Look at contempt's row: precision 1.00 but recall 0.15. It only says "contempt" when it's very sure, so it's right when it guesses, but it misses 85% of them. That's the rare-class problem made concrete, not a bug.

The mistakes it does make are human ones: sad confused with neutral, fear with surprise. And there's a ceiling here worth naming. Around 80% is close to the best a small model gets on FER+; even large, ensembled networks top out near 84–85%. The reason is the data itself — the ten annotators often disagreed, so past a point there is no single "right" answer to learn. Emotion from one grainy face is genuinely hard, for machines and for people.

Try the model on your own face

This is the exact model from above, running entirely in your browser through ONNX Runtime Web. Nothing is uploaded — the video never leaves your machine. Allow the camera, keep your face in the box, and the bars update in real time.

Live emotion recognition — your trained CNN, in the browser.

Everything, end to end

The whole project is on GitHub — the notebook that goes from raw data to a trained model, the saved weights, and this browser demo. It runs top to bottom, so you can reproduce every number above.