Substrate

Composition matters as much as scale. Code in the mix improves reasoning on tasks that are not code, and a small fraction of carefully curated text can outweigh a large fraction of crawl. The filtering stack usually runs language identification, then deduplication (exact first, then near-duplicate), then quality classification, then safety filtering, then decontamination against known benchmarks. Ordering effects are real but poorly understood. Two consequences travel with the model for the rest of its life. There is a knowledge cutoff: nothing after the collection date exists to it. And there is inherited distribution: whose writing is over-represented in the corpus becomes whose assumptions are over-represented in the output. Neither can be fixed afterwards by prompting. Licensing and consent are unresolved and actively litigated. That is an open question rather than a settled practice, and this site does not pretend otherwise.

The standard method is byte-pair encoding. Start from individual bytes, repeatedly merge the most frequent adjacent pair, and stop at the target vocabulary size. The merges are learned from the training data, so the vocabulary reflects what that data contained. A few consequences show up constantly. Leading spaces are usually part of the token, so " the" and "the" have different IDs, which is why a stray space in a prompt can change the output. Numbers fragment unpredictably, and the model never sees the digits as a quantity, which contributes to arithmetic errors. Languages that were under-represented in training cost two to three times more tokens per word, so they are literally more expensive to use and they fill the context window faster. Character-level tasks like counting letters or reversing a string are hard for the same reason: the model never sees characters at all. Tokenization is also a reliability surface. Unusual byte sequences produce tokens the model saw almost never during training, and behaviour on those is poorly characterised.

The initial embedding is context-free. One vector per token ID, identical every time. Context is added by the layers above, so by the final layer the representation of "bank" in a river sentence differs from "bank" in a finance sentence. That distinction between static input embeddings and contextual hidden states causes most of the confusion when people talk about "the embedding" of a word. Similarity is measured by cosine, the angle between two vectors, rather than by straight-line distance. Magnitude tends to encode frequency rather than meaning, so ignoring it is deliberate. The famous analogy arithmetic works partially and unevenly. It is a real property of the space and it gets oversold: the nearest result is often the input word itself, which is why demonstrations quietly exclude it. The same idea scales up from tokens to whole passages, which is what makes retrieval possible later on.

Each transformer layer has two parts. Attention mixes information between positions, and a feed-forward network transforms each position independently. Most of the parameters sit in the feed-forward blocks: about two thirds in the older designs, and closer to four fifths once grouped-query attention shrank the attention side. That is where most factual association appears to live, distributed across many parameters rather than localised in any one. Precision is a real lever. The same model at 16, 8 or 4 bits per parameter differs in size and speed by multiples, with quality loss that stays small until it suddenly does not. Mixture-of-experts architectures hold many parameters but activate only a fraction for each token. That decouples storage cost from compute cost, and it makes parameter counts a poor basis for comparing two models.

Backpropagation computes the gradient of the loss with respect to every parameter. An optimizer then applies it, scaled by a learning rate, with momentum and adaptive per-parameter scaling. Training runs in batches, and the learning rate follows a schedule that warms up and then decays. Loss falls fast, then logarithmically. The last increments are enormously expensive, which is most of why frontier training runs cost what they do. Capabilities do not appear smoothly. Some tasks sit at chance for a long time and then improve sharply, which makes progress genuinely hard to predict from a loss curve. People watch loss curves obsessively anyway, because a diverging run can waste weeks of compute before anyone notices. The whole process is one optimization problem. It is very large and very expensive, and conceptually it is simple.

Scaling laws relate loss to data, parameters and compute as a power law over many orders of magnitude, and they hold well enough to plan a training run against. The compute-optimal ratio between data and parameters has been revised more than once, which is worth remembering when a current figure is quoted as settled. Emergence is contested rather than mysterious. Part of the sharpness is real and part is an artifact of grading: a task scored pass-or-fail jumps when partial competence crosses the threshold, while the underlying improvement was smooth all along. How much of it is measurement is an open question. The practical consequence is uncomfortable for everyone involved. You can predict what a run will cost and roughly how well it will predict text. You cannot predict what it will be able to do, which means capability is discovered after the money is spent.

What ships is a set of tensors, a config describing the architecture, and a tokenizer vocabulary. There is no database, no index and no runtime state. Serving it means loading it into memory and running arithmetic against it. The same file behind an API and on a laptop produces the same distribution given the same input and settings, which is what makes the API a convenience rather than a different machine. Fine-tuning creates a new file, or a small companion file of adjustments called an adapter. It does not edit the original. This is why "the model remembers me" is always wrong, and always describes something else happening outside the file: a transcript being resent, a note being stored, a document being retrieved. Every one of those is a program somebody wrote, and every one of them shows up in the stages that follow.

There are two stages. Supervised fine-tuning teaches the response format and the assistant role. Preference optimization then adjusts the model toward outputs humans rated higher, historically through a learned reward model and reinforcement learning, and increasingly through direct methods that skip the separate reward model. A third stage trains against verifiable outcomes: did the code run, is the proof valid. That is what produces long deliberate reasoning traces. Post-training is where the personality, the refusal behaviour and the formatting habits come from. It accounts for most of the visible difference between two models built on similar pretraining. It is also fragile. Heavy alignment training can measurably reduce capability on some tasks, and that trade-off is not solved.