How AI models are trained: from raw text to a system that helps

Nobody writes a language model. That's the first thing to understand, and it's stranger than it sounds. A large language model is not programmed with rules, facts, or instructions the way ordinary software is. It is grown, trained, in stages, from an enormous quantity of text into a system that behaves as though it understands language, follows instructions, and knows things. The behaviour you experience when you chat with one is the end product of a long pipeline, and almost everything interesting about these systems, why they're capable, why they hallucinate, why they refuse some requests, why they cost tens of millions of dollars to build, traces back to how they were trained.

This is that process, explained in full: the modern pipeline that turns raw text into a helpful assistant. It comes in stages, each doing a distinct job, and understanding them in order dissolves most of the mystery around what these systems are. It's also a story that changed recently, the tidy "pretrain, then RLHF" recipe that defined the early 2020s has, by 2026, become a modular multi-stage stack with new techniques at almost every step. We'll walk the pipeline as it actually is now, and note where the ground moved.

Stage 1: Pretraining, learning language from the firehose

Everything starts with pretraining, and this is the stage that does the heavy lifting, where the model acquires essentially all of its knowledge and its raw command of language. It's also, by far, the most expensive: pretraining a frontier model in 2026 costs somewhere in the range of tens to low hundreds of millions of dollars in compute alone.

The mechanism is deceptively simple. Take a staggering quantity of text, a large fraction of the public internet, books, code, cleaned and filtered, measured in trillions of tokens, and train a transformer to do one thing: predict the next token. Show it "The capital of France is ___" and train it to predict "Paris." Do this over and over, across trillions of examples, adjusting the model's billions of parameters each time to make its predictions a little more accurate. That's it. That's the entire objective: next-token prediction, at unimaginable scale.

What's remarkable, and surprising, even to the researchers who built the first ones, is what falls out of that simple objective. To predict the next token well across the entire internet, the model is forced to learn grammar, facts, reasoning patterns, the structure of code, translations between languages, and countless other regularities, because all of those help it predict better. Knowledge enters the model here, as a side effect of compression: the best way to predict text about France is to internalise facts about France. This is also why a model's knowledge cutoff exists, it knows what was in its pretraining data and nothing after, because that's the only place its knowledge came from. It didn't "forget" recent news; it never saw it.

The product of pretraining is called a base model, and here's the key thing: a base model is not a chatbot. It's a raw text-completion engine. Ask it a question and it might continue with more questions, because on the internet, questions are often followed by more questions. It has vast knowledge and language ability but no notion of being a helpful assistant. It's brilliant and unusable, a mind with no manners. Everything after pretraining is about shaping that raw capability into something useful, and it's remarkable how little additional data that takes compared to the ocean of pretraining.

One important 2026 wrinkle: pretraining is bumping against a data wall. The supply of high-quality human-written text is finite, and the largest models have essentially consumed it. This is why synthetic data, text generated by other models, has become central, and why scaling laws, which promised smooth improvement from simply adding more data, are getting harder to ride on data alone. Labs increasingly also add a mid-training phase here, continuing to train the base model on higher-quality or domain-specific data to sharpen it before the shaping begins.

Stage 2: Supervised fine-tuning, learning to be helpful

The first shaping stage is supervised fine-tuning (SFT), and its job is to teach the base model the format of being an assistant: that when a human asks something, you answer it, helpfully and directly.

The method is straightforward fine-tuning. You take the base model and continue training it, but now on a much smaller, carefully curated set of example conversations, high-quality demonstrations of the behaviour you want: a question, and an ideal helpful answer; an instruction, and a good response. These examples are written by humans or, increasingly in 2026, generated synthetically and filtered for quality (instruction tuning is the name for this when the examples are instructions). The model learns, from these demonstrations, to adopt the shape of a helpful respondent: when it sees a question, it now produces an answer rather than more questions.

What's striking is how few examples this takes, thousands to tens of thousands, versus the trillions of tokens of pretraining. That's because SFT isn't teaching the model knowledge or ability; those already exist from pretraining. It's teaching format, how to deploy what it already knows in the shape of a conversation. The capability was there all along; SFT just points it in the right direction. After this stage you have a model that's usable as an assistant. But it's not yet aligned with the finer points of what people prefer, which is the next stage's job.

Stage 3: Preference alignment, learning what people prefer

SFT teaches the model to answer, but not which of two reasonable answers is better, more helpful, more honest, safer, better-toned. That subtler shaping comes from preference alignment, and this is the stage that changed the most between the early 2020s and 2026.

The original method, and still the one most people have heard of, is RLHF, reinforcement learning from human feedback. The idea: show humans pairs of model responses and ask which is better; train a separate reward model to predict those human preferences; then use reinforcement learning (specifically an actor-critic method called PPO) to optimise the assistant to produce responses the reward model scores highly. It's simple and it works, RLHF is what first made models feel aligned and well-behaved, and every early frontier model relied on it.

But RLHF is hard: it requires training three separate models (the assistant, the reward model, the reference), managing a finicky RL loop, and careful tuning. That difficulty drove the biggest shift in the modern pipeline. By 2026, classic RLHF is often replaced by Direct Preference Optimization (DPO) and its relatives, which achieve the same goal, aligning the model to preferred responses, but reframe it as a simpler classification problem: the model sees (chosen, rejected) response pairs and directly learns to make the chosen one more likely, with no separate reward model and no RL loop. The result is comparable to RLHF at a fraction of the compute cost and far more stability. DPO and its variants have become a default because they get most of RLHF's benefit with much less pain.

There's also a notable variation on where the preferences come from. Anthropic's Constitutional AI modifies this stage by generating preference judgments from a written set of principles (a "constitution") rather than collecting every judgment from human labellers, using the model itself, guided by principles, to produce the training signal. It's not a replacement for the alignment stage but a change to how its data is made, and it's part of a broader move toward scaling alignment without scaling human labelling.

Whatever the method, the output of this stage is a model aligned with human preferences: helpful, appropriately honest, and able to refuse harmful requests. This is also, notably, the stage where some of the incentive problems behind hallucination get baked in, if the preference data rewards confident answers over honest uncertainty, the model learns to bluff.

Stage 4: Reasoning training, the 2026 defining stage

Here is the stage that barely existed a few years ago and now defines the frontier: training models to reason.

The breakthrough was the discovery that you can dramatically improve a model's performance on hard problems, math, code, logic, by training it, via reinforcement learning, to produce long chains of thought before answering, and rewarding it based on whether the final answer is correct. The critical enabler is verifiable rewards: for math and code, you don't need a human or a learned reward model to judge the answer. You can check it automatically. A math answer is right or wrong; a unit test passes or fails. This approach, often called RLVR (reinforcement learning with verifiable rewards), removes the human bottleneck entirely and eliminates a whole class of gaming, because the reward is ground truth, not a prediction of human opinion.

The results reshaped the field. Models trained this way learn, on their own, to generate long reasoning traces, to check their own work, to backtrack when a line of attack fails, behaviours that emerge from the training rather than being explicitly programmed. This is what powers the reasoning models that "think" before answering. Every major reasoning model since late 2024 uses some form of verifier-driven RL, and it's become a standard pipeline stage: after alignment, train on verifiable problems to build reasoning. (It's not a universal good, as covered in our piece on why models hallucinate, extended reasoning can increase fabrication on open-ended factual questions even as it helps on math. Reasoning training is powerful, not magic.)

The modern pipeline, assembled

Put the stages together and the modern recipe is a modular stack, not a single process, and different models mix the modules differently:

Pretraining (learn language and knowledge from trillions of tokens) → optional mid-training (sharpen on higher-quality data) → supervised fine-tuning (learn the helpful-assistant format) → preference alignment via DPO or RLHF (learn what people prefer, refuse harm) → reasoning training via RLVR (build step-by-step problem-solving on verifiable tasks). Some models branch here into separate "instruct" and "reasoning" variants from a shared base.

The key mental shift from the textbook story is that this is now modular and evolving. "Pretrain, then RLHF" was the recipe of 2022. By 2026 it's a configurable pipeline of specialised stages, each with its own best-in-class technique, mixed to taste, DPO here, RLVR there, synthetic data throughout, constitutional methods for alignment. The stages are stable in purpose (knowledge, then format, then preference, then reasoning) even as the methods inside them turn over rapidly.

Why the pipeline explains almost everything

Once you see the pipeline, a lot of otherwise-puzzling things about AI models click into place, and this is why it's worth knowing.

Why do models hallucinate? Because knowledge lives fuzzily in the pretrained weights, and preference alignment rewarded confident answers over "I don't know." Why is there a knowledge cutoff? Because knowledge enters only during pretraining, and pretraining data ends at a date. Why do models sometimes get worse at one thing after being trained to be better at another? Because fine-tuning can cause catastrophic forgetting, overwriting earlier capability. Why can't the model just learn your company's data by "reading" it? Because learning requires a training run; that's what retrieval is for, sidestepping training entirely. Why do the best small models punch above their weight? Often distillation, training a small model on a large one's outputs, which is a training-pipeline choice. Why do models refuse harmful requests? Because alignment trained them to. Nearly every characteristic behaviour of an AI system is a fingerprint of some stage of how it was trained.

The single most important thing the pipeline reveals is the division of labour: capability and knowledge come from pretraining; behaviour and helpfulness come from the shaping stages after. A model is a brilliant, knowledgeable, unusable base, painstakingly shaped into something that helps, and both halves matter. The raw intelligence without the shaping is unusable; the shaping without the raw intelligence has nothing to work with.

The short version

An AI model is grown, not written. Pretraining teaches it language and knowledge by predicting the next token across trillions of tokens of text, producing a brilliant but unusable base model. Supervised fine-tuning teaches it the format of being a helpful assistant from a small set of good examples. Preference alignment, classic RLHF, or increasingly the simpler DPO, teaches it which answers people prefer and how to refuse harm. And reasoning training with verifiable rewards, the defining stage of 2026, teaches it to think step by step on problems whose answers can be checked. Each stage builds on the last, and none can be skipped.

a model's knowledge and ability come from pretraining; its helpfulness, honesty, and reasoning are shaped in afterward, so nearly everything a model does, well or badly, is a fingerprint of how it was trained. Understand the pipeline, and the model stops being a mysterious oracle and becomes something you can reason about: a system whose every strength and flaw was installed, on purpose, one stage at a time.

Common questions

How are large language models trained? In stages. First, pretraining: the model learns language and knowledge by predicting the next token across trillions of tokens of text, producing a raw "base model." Then supervised fine-tuning teaches it to behave like a helpful assistant using curated example conversations. Then preference alignment (RLHF or the simpler DPO) teaches it which responses people prefer and how to refuse harmful requests. Finally, reasoning training with verifiable rewards teaches it to think step by step. Each stage builds on the last and none can be skipped.

What is the difference between pretraining and fine-tuning? Pretraining is the massive, expensive first stage where the model learns language and essentially all its knowledge by predicting the next token across trillions of tokens. This is where capability comes from. Fine-tuning is the much smaller, cheaper shaping that comes after, adjusting the model on curated examples to teach behaviour: how to be a helpful assistant, what tone to use, what to refuse. Pretraining builds the raw mind; fine-tuning gives it manners. Capability from pretraining, behaviour from fine-tuning.

What is RLHF and is it still used? RLHF (reinforcement learning from human feedback) aligns a model with human preferences: humans rank pairs of responses, a reward model learns to predict those rankings, and the model is optimized to score well. It's what first made models feel aligned. In 2026 it's often replaced by simpler alternatives like DPO (Direct Preference Optimization), which achieves similar alignment without a separate reward model or reinforcement learning loop, at much lower cost. RLHF's ideas remain central even where its exact original method has been superseded.

What is a base model? A base model is the product of pretraining alone, a raw next-token predictor with vast knowledge and language ability but no notion of being a helpful assistant. Ask a base model a question and it might continue with more questions, because it just completes text in statistically plausible ways. It's brilliant but unusable as a chatbot. The fine-tuning and alignment stages after pretraining are what turn a base model into the helpful assistant you actually interact with.

Why do AI models have a knowledge cutoff? Because a model's knowledge enters almost entirely during pretraining, and pretraining data ends at a certain date. The model doesn't "forget" more recent events, it never saw them, because they weren't in the training data. This is a feature of how training works, not a bug, and it's why retrieval (RAG) and web search exist: to give models access to current information at question time without retraining, since adding new knowledge to the weights themselves requires an expensive new training run.

What is RLVR (reinforcement learning with verifiable rewards)? RLVR is the reasoning-training method that defines frontier models in 2026. Instead of rewarding a model based on predicted human preference, it rewards based on whether the answer is verifiably correct, a math answer that checks out, a unit test that passes. This lets models train on millions of automatically-verified problems with no human bottleneck and no reward gaming, since the reward is ground truth. Models trained this way learn to produce long reasoning chains and self-check their work, which is what powers "thinking" reasoning models.

Why is training AI models so expensive? Almost all the cost is in pretraining, which requires running enormous computations over trillions of tokens to adjust billions of parameters, costing tens to hundreds of millions of dollars in compute for a frontier model. The shaping stages afterward (fine-tuning, alignment) are far cheaper because they use much less data. This cost asymmetry is why most organizations never pretrain their own models; they take an existing base or open model and fine-tune it, which is thousands of times cheaper than pretraining from scratch.