How LLMs Work
What an LLM actually is
A Large Language Model is a neural network — specifically, a transformer — trained to predict the next token given all previous tokens. That's the entire training objective. Predict the next word, repeatedly, at massive scale.
The "large" part: GPT-3 has 175 billion parameters. GPT-4 is estimated at over 1 trillion. These parameters are the weights learned during training — the compressed statistical knowledge of the training corpus.
When you prompt an LLM, you're not querying a database. You're running a very large matrix multiplication chain that outputs a probability distribution over the next possible token.
Pre-training: reading the internet
Pre-training is where LLMs get their knowledge. The model sees hundreds of billions of tokens from the web, books, code, and academic papers. For each position in a sequence, it predicts the next token and updates weights based on how wrong it was.
The training signal is self-supervised — no human labels needed. The label is always "the next token that actually appears in the text."
What the model learns:
- Grammar and syntax (unavoidable side effect of next-token prediction)
- Facts (Paris is the capital of France appears millions of times)
- Reasoning patterns (if-then chains, logical arguments appear in text)
- Code (the same prediction objective works on Python as on English)
Pre-training costs tens to hundreds of millions of dollars in compute. GPT-4's training run is estimated at over $100M.
Context windows
The context window is how many tokens the model can "see" at once when generating a response. Everything outside the context window is invisible.
GPT-4: 128,000 tokens (roughly 100,000 words)
Claude 3.5: 200,000 tokens
Gemini 1.5 Pro: 1,000,000 tokens
Longer context windows let models process entire codebases, long documents, or extended conversations. But attention is O(n²) in sequence length, so longer windows cost significantly more compute.
Within the context window, not all positions are equal. Models pay more attention to the beginning and end of the context (primacy and recency effects). Long-context models still struggle with information buried in the middle.
Fine-tuning and RLHF
A pre-trained LLM knows a lot, but it will also confidently generate harmful content, make things up, and be generally difficult to use as an assistant. Pre-training alone produces a text predictor, not a helpful AI.
Supervised Fine-Tuning (SFT): Show the model thousands of example conversations where a human wrote the ideal response. Train the model to mimic these responses.
RLHF (Reinforcement Learning from Human Feedback):
1. Generate multiple responses to a prompt
2. Human raters rank the responses from best to worst
3. Train a "reward model" to predict human preference
4. Use reinforcement learning (PPO) to fine-tune the LLM to maximize the reward model's score
RLHF is why ChatGPT feels different from raw GPT-3. It's the same base model, trained to be helpful, harmless, and honest rather than just statistically likely.
RLHF trade-off: Fine-tuning for helpfulness can reduce raw capability. Models can become overly cautious, sycophantic, or forget knowledge that wasn't in the fine-tuning set.
Inference: how generation actually works
When you send a prompt, the model doesn't "think" and then output an answer. It generates one token at a time, feeding each generated token back in as context for the next prediction.
"The capital of France is" → generates "Paris" → "The capital of France is Paris" → generates "." → done.
Temperature controls randomness. Temperature 0 = always pick the highest-probability token (deterministic). Temperature 1 = sample from the distribution. Temperature > 1 = more random, less coherent.
Top-p sampling (nucleus sampling): only sample from the smallest set of tokens whose cumulative probability exceeds p. Reduces the chance of sampling very low-probability garbage tokens.
Beam search: maintain multiple candidate sequences simultaneously, pick the best. Common in translation tasks, less common in chat.
Hallucination is not a bug — it's a feature misused
Tokens, not words
Everything above uses "token" deliberately. Tokens ≠ words. Using OpenAI's tokenizer:
"ChatGPT is great" = 4 tokens
"Supercalifragilistic" = 5 tokens
"日本語" = 3 tokens
Rules of thumb:
- 1 token ≈ 0.75 English words
- 1,000 tokens ≈ 750 words ≈ 3 pages
- 100,000 tokens ≈ 75,000 words ≈ a full-length novel
API pricing is per token. Context window limits are in tokens. Knowing token counts matters for both cost and performance.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.