How a weight is made

A weight is one number. Watch one get made from the sentence "My puppy name is River. River likes Ruby.", then watch it get used.

Taste it→How far off?→Add a pinch→Taste again

Training works like getting the salt right in a curry. The closer it gets, the smaller the pinch.

1 · Training makes it

The number for "my → puppy"

This number says how strongly the model expects puppy after my. It starts as a random number, so the model knows nothing.

my → puppy
–
0%striped = missing100% (the goal)
2 · Saving writes it down

Like writing down the recipe

When training is done, the number is saved in the model's file. From now on it never changes.

Nothing saved yet.
3 · A prompt reads it

Pick a word, the model looks up its numbers

Pick the last word of a prompt. The model reads that word's saved numbers, turns them into chances, and the biggest wins.

Real models make every weight this way: start random, compare the guess with the right answer, nudge the number a little in the direction that shrinks the gap, repeat. They just do it for billions of numbers at once, over trillions of words.

In a big model most numbers sit deep inside, so each one's share of the mistake is worked out by tracing the error backwards through the model (backpropagation). The nudge rule is the same idea.