In the last post I tried to explain how an LLM gets built. Billions of numbers, adjusted one fraction at a time, until structure emerges from prediction pressure. Circuits form. Clusters of meaning self-organize.
But that post ends where the interesting part begins. Now the model exists. The weights are frozen. Training is done.
Now we type something and hit enter. What actually happens?
This is the post I wish I’d had when I started using these tools.
The Forward Pass
We type a message, Text gets chopped into tokens.
Subword chunks, not full words. “Understanding” becomes something like ["under", "standing"]. Your message might be 20 words but 30+ tokens.
Those tokens flow forward through the model’s layers. Every layer transforms the representation. The attention mechanism lets each token look back at every other token in the context and decide what’s relevant.
The weights don’t change during this process. They’re frozen from training. The model is just running, applying its learned patterns to your specific input.
What comes out is a list.
Lets Say we type: "Write a short paragraph about Kafka vs RabbitMQ"
The model tokenizes that, processes it through every layer, and has to pick the very first token of its response.
To do that, it computes a score for every token in its vocabulary.
What’s a vocabulary?
The vocabulary is the fixed list of every token the model knows, built before training using byte-pair encoding on a massive text corpus that LLM companies have scraped off the internet.
For GPT-2, that’s 50,257 tokens. For newer models it’s larger, often 100k+.
The output is a probability distribution across that entire vocabulary, every time, for every single token it generates.
We are going to use GPT-2 as an example to explain the concept
For that first token, the raw scores (logits) might look something like this:
Token 8,527 ("When"): 0.1263
Token 16,401 ("Kafka"): 0.0891
Token 3,198 ("The"): 0.0734
Token 11,045 ("Both"): 0.0622
Token 23,189 ("Apache"): 0.0418
Token 1,550 ("In"): 0.0387
Token 42,007 ("Choosing"): 0.0095
Token 7,904 ("Message"): 0.0071
Token 33,421 ("While"): 0.0068
Token 50,012 ("ĠðŁ"): 0.0000003
Token 831 (" Q"): 0.0000001
...
[50,246 more entries trailing into the decimals]
Every generation step. Fifty thousand scores. The model doesn’t “think of” the top five and pick one.
It produces all 50,257 simultaneously and the sampling process decides which one wins. Most of that list is near-zero noise.
Tokens like emoji fragments and random punctuation that have no business starting a paragraph about message brokers. But they’re scored anyway. Every time.
This is the fundamental object we’re manipulating every time we use these tools.
A probability distribution over the entire vocabulary, shaped by everything the model has seen so far in the context window.
Let’s hang onto that mental image. It will make everything else in this post make sense.
The Dice Roll in Practice
Other post covered temperature conceptually. Low temp means predictable, high temp means creative. But knowing how it works changes how we use it.
The model produces those raw scores (logits) for all 50,257 tokens. Temperature divides those scores before they get converted to probabilities. That division matters.
Lets use our Kafka vs RabbitMQ prompt and trace what happens.
Low temperature (0.2): Stick to the spec
The division amplifies the gaps between scores. “When” was already the top pick, and after low-temp scaling it dominates. The model opens with “When” almost every time. Run it five times:
Run 1: "When comparing Kafka and RabbitMQ, the key distinction lies in..."
Run 2: "When choosing between Kafka and RabbitMQ, it's important to..."
Run 3: "When evaluating message brokers, Kafka and RabbitMQ represent..."
Run 4: "When comparing Kafka and RabbitMQ, the key distinction lies in..."
Run 5: "When choosing between Kafka and RabbitMQ, the fundamental..."
Nearly identical openings. The first token barely varies, and that initial choice constrains everything that follows
Runs 1 and 4 might be word-for-word identical for the first 20 tokens before the dice diverge.
High temperature (1.0): Creative
The division shrinks the gaps. “When” is still the most probable, but “Kafka,” “Both,” “Apache,” “Choosing” all get a real shot. The outputs sprawl:
Run 1: "Both systems handle messaging but their philosophies diverge..."
Run 2: "Kafka treats the log as the fundamental abstraction..."
Run 3: "Choosing between these two usually comes down to whether..."
Run 4: "Apache Kafka and RabbitMQ solve overlapping problems from..."
Run 5: "In the messaging landscape, Kafka and RabbitMQ occupy..."
Same model. Same weights. Same prompt. Five different opening tokens, five different directions.
Each choice cascades. Once the model starts with “Kafka treats the log,” the next token distribution shifts entirely compared to starting with “Both systems handle.”
Temperature = 0: Greedy decoding. Always pick the highest-scoring token. Completely deterministic, same input, same output, every time. No dice roll at all.
Then there’s the filtering that happens before the roll. Top-k says “only consider the k highest-scoring tokens, zero out everything else.” Top-p (nucleus sampling) says “start from the top and keep adding tokens until their cumulative probability reaches p%, then zero out the rest.” Most production systems use some combination of all three. (If you want the full technical breakdown of these decoding methods, Hugging Face’s walkthrough is excellent.)
This is why “regenerate” gives us a different response. Same weights, same context, same list of 50,257 scores. Different roll of the dice. The terrain is identical. The path through it changes.
And here’s the practical bit: when you set temperature to 0.2 for a coding task and 0.8 for brainstorming, we are not changing the model. nor accessing different capabilities. we are changing how aggressively it samples from the same probability distribution.
The model “knows” the same things at both temperatures. we are just deciding how much we trust its top picks.
This is why modern agentic coding tools like Cursor/Cline/Codex have separate modes for planning and coding/debugging.
Planning needs to explore options, consider architectures, think laterally. That’s higher temperature territory.
Writing the actual code using the plan created by the planning mode, needs to be precise and deterministic. That’s low temperature.
Same model behind both modes. Different sampling strategy for different phases of the work.
Where the knobs actually are
In ChatGPT or Claude through the web UI, we can’t control the temperature directly. The provider picks a default.
With the Claude API, we get temperature (0.0 to 1.0, defaults to 1.0), top_k, and top_p. Anthropic’s guidance is to just use temperature and leave the others alone.
With Google’s Gemini, we get temperature controls directly in the AI Studio UI. No API needed, just a slider. Their range goes from 0.0 to 2.0.
Temperature 0.5 on Claude and temperature 0.5 on Gemini don’t produce the same behavior.
Each provider trains and tunes differently, so the same number produces different sampling characteristics. It’s the same concept across all of them, but we can’t just copy settings between providers and expect identical results.
System Prompts as Activation Space Anchoring
Before I started going down this rabbit hole, I assumed system prompts worked like config flags.
“Set the model to be a Python expert.” “Tell it to be concise.” Flips a switch in the model and changes the behavior. I think most people using these tools have that same mental model.
Turns out I was wrong. And understanding what’s actually happening made me noticeably better at using these tools.
A system prompt is text. It gets tokenized and fed into the model as the first tokens in the context window. Those tokens flow through the same layers as everything else. They produce activations, patterns of neural activity inside the model. And those activations influence every token that comes after.
Check the galaxy map from Anthropic’s feature visualization, where concepts cluster into neighborhoods
Code near code, legal language near legal language, casual conversation near casual conversation
The system prompt doesn’t tell the model which neighborhood to visit. It starts the model in that neighborhood.
When you write "You are a senior Python developer who writes production code with proper error handling, type hints, and logging"
every one of those tokens activates features in the model. “Senior” pulls toward experienced patterns.
“Production” pulls toward robustness. “Error handling,” “type hints,” “logging” each activate their own clusters.
Those activations become part of the context. Every subsequent token the model generates is influenced by them because the attention mechanism lets every new token look back at the system prompt tokens.
(For Claude API, Anthropic has documentation on how system prompts work at the implementation level.)
The system prompt seeds the context window with tokens that bias which internal features and clusters activate. It pulls the model into a specific region of its representation space. this is activation space anchoring.
(This isn’t just theory. we can literally steer GPT-2’s behavior by adding activation vectors into its forward pass. Add a “wedding” vector and the model talks about weddings. Add an “anger” vector and it gets hostile. The activations are the steering mechanism, and system prompts are doing a version of the same thing through natural language.)
And this connects directly to the probability list. The system prompt doesn’t add new tokens to the vocabulary. It doesn’t unlock hidden capabilities.
What it does is reshape the probability distribution over those same 50,257 tokens.
Tokens related to the system prompt’s domain get boosted. so it will assign high probability to tokens in that subjects domain
Take our Kafka vs RabbitMQ prompt again. Without a system prompt, the first-token distribution had “When” on top, “Kafka” and “Both” trailing behind, a generic opening for a generic comparison.
Now add a system prompt: "You are a senior distributed systems architect. Prioritize throughput, partition tolerance, and operational tradeoffs. Be direct."
The same prompt. But those system prompt tokens have been flowing through the model’s layers, activating features related to distributed systems, performance, architecture. By the time the model gets to our question, the probability landscape has shifted:
Token 16,401 ("Kafka"): 0.1534 (was 0.0891)
Token 3,198 ("The"): 0.0812 (was 0.0734)
Token 6,571 ("At"): 0.0498 (new in top 10)
Token 11,045 ("Both"): 0.0411 (was 0.0622)
Token 8,527 ("When"): 0.0389 (was 0.1263, dropped hard)
Token 19,888 ("From"): 0.0285 (new in top 10)
Token 23,189 ("Apache"): 0.0271 (was 0.0418)
“When,” the safe essay-style opener, dropped from first place to fifth. “Kafka” jumped to the top. The model is more likely to lead with the technical substance rather than a comparison framework. That "Be direct" token cluster suppressed the hedging openers. The distributed systems context boosted tokens that lead to architectural analysis.
Same vocabulary. Same 50,257 entries. Different weights across the list.
Simplified interactive visualization of Activation Space Anchoring
Temperature + System Prompt: Two Knobs, One Process
Once you see it this way, temperature and system prompts stop being separate concepts.
The system prompt shapes which probabilities are high and which are low. It sculpts the distribution. Boosts code tokens, suppresses casual ones, or whatever the prompt content biases toward.
Temperature controls how strictly the model follows that shaped distribution.
Low temperature means “stick to what the system prompt is pushing you toward.”
High temperature means “the system prompt set a direction, but feel free to wander.”
They’re two knobs on the same process. One shapes the probability field. The other controls how tightly the model walks along its ridges.
MoE Routing: When the Architecture Gets Involved
Some models take this further. Mixture of Experts (MoE) architectures, used in models like Gemini and DeepSeek, don’t activate all their parameters for every token. They route each token through a subset of specialized “expert” subnetworks. (Hugging Face has a solid explainer on how MoE works with a full architecture breakdown.)
tldr;
In a MoE model, the system prompt tokens flow through the network and produce hidden states, just like in a dense model. But the way they influence routing is indirect, and it matters to get this right.
The router itself is stateless. It’s a simple feed-forward layer that looks at one token’s hidden state and decides which experts to use. It has no memory of what came before. So the system prompt tokens don’t “tilt” the router or bias it over time.
What actually happens is the Attention mechanism does the work first. When a token from your actual question (say “Kafka”) is being processed, it attends back to the system prompt tokens (“You are a distributed systems architect”).
That attention pulls system-prompt context into the current token’s hidden state vector. By the time that enriched “Kafka” vector reaches the MoE layer, it looks different than it would without the system prompt. The router sees that specific vector, evaluates it, and routes it to the experts that match. A “Kafka” vector colored by distributed systems context gets routed differently than a “Kafka” vector colored by literary analysis context.
It’s not a clean “wake up the code expert” signal. It’s per-token and indirect. The system prompt infects each new token through Attention, and that infected representation is what the router evaluates.
The effect is real, but the mechanism is Attention doing the heavy lifting before the router ever sees the token.
This is very similar to the activation anchoring principle, but operating at an additional architectural level. Not just biasing which features activate within a single network, but biasing which sub-networks get used at all.
Why Models Drift in Long Conversations
This one drove me nuts before I understood the mechanism.
we write a careful system prompt. The model follows it perfectly for 10 messages. By message 20, it’s drifting. The tone shifts.
It starts complimenting you. It forgets constraints you set. With some models, the anti-sycophancy instructions you wrote might as well not exist after enough back-and-forth.
The architecture explains exactly why.
Attention has a cost that scales with context length. As the conversation grows, each new token has more previous tokens to attend to. The system prompt tokens are still there, they haven’t been deleted, but they’re now a small fraction of a much larger context window.
Think of it like a voice in a growing crowd. Your system prompt is a person at the front of the room speaking clearly. When there are 10 people in the room, everyone hears this person fine. When there are 500 people all talking, that original voice gets harder to pick out.
Transformers don’t inherently know word order, so they use positional encodings (like RoPE, Rotary Position Embedding) to inject position information into each token.
These encodings bias the attention mechanism to favor tokens that are physically closer. As the conversation gets longer, the physical distance between the current token and the system prompt grows.
Now when we Combine that distance penalty with the fact that recent back-and-forth dialogue we built up in the chat, the system prompt’s anchoring effect fades.
And what fills the gap is the model’s base personality. The behaviors baked in during RLHF and preference tuning.
The agreeable, helpful, slightly sycophantic tendencies that training optimized for. The system prompt was overriding those tendencies, but as its influence weakens, the base behavior seeps back through.
This is why context window isn’t just a memory constraint. It’s a behavioral stability constraint.
A model with a 128k context window doesn’t just remember more, it maintains system prompt influence over a longer conversation.
( “Lost in the Middle” Paper shows language models perform best when relevant information is at the beginning or end of the context, and significantly worse when it’s buried in the middle. system prompt sits at the very beginning, which helps, but distance penalty applies)
Practical Implications
Dense system prompts beat fluffy ones.
Length isn’t the problem. Anthropic’s own default system prompt for Claude is thousands of tokens long, and it works. A 2,000-token prompt packed with dense architectural constraints, few-shot examples, strict schemas, and specific behavioral rules.
This creates a massive anchor in the context that practically forces the model into a specific behavioral subspace.
But a 2,000-token prompt full of vague running sentences (“Be a helpful, friendly, synergistic assistant who always puts the user first”) is actively sabotaging the prompt and just burning tokens and a little hole in your wallet and warming our planet.
Every token in the system prompt must earn its keep. The failure mode isn’t “too long,” it’s “too much noise.” Contradictory instructions, redundant phrasing, and generic filler all dilute the signal of the tokens that actually matter.
Domain context is activation anchoring.
When we paste a code file, an API schema, or a data model into the context, we are not just “giving the model information.” Its flooding the context with domain-specific tokens that bias the entire activation landscape.
This is why RAG (Retrieval-Augmented Generation) is popular. Not just because the model “reads” the retrieved documents, but because those documents’ tokens reshape the probability distribution toward domain-relevant outputs.
Temperature stacking with system prompts.
Now we can be deliberate: use a tight system prompt to sculpt the distribution, then use temperature to control variance within that sculpted space.
Tight prompt + low temp for implementation.
Tight prompt + higher temp for exploring design alternatives. Same anchor, different sampling discipline.
Mitigations
Refresh the system prompt in long conversations. when you are 30 messages deep and the model is drifting, restating the key constraints will re-anchor the model. we are injecting fresh system-prompt-like tokens closer to the model’s current attention window, boosting their influence relative to the stale tokens at the beginning.
Use spec-based development and write skills. Every modern agent supports them. A spec is a dense, structured document that front-loads context.
Skills are reusable instruction sets that get injected into the system prompt. Both are mechanisms for packing the context window with high-signal tokens that keep the model anchored to what we actually want. I wrote about this workflow in a previous post.
Same Patterns, Different Layer
At the inference layer, the mechanism is different but the shape is the same.
We write a prompt. Those tokens create activation patterns. Those patterns bias a probability distribution. Sampling selects from that distribution. The output feeds back in and the loop continues. Simple operations, iterated, producing behavior that looks like understanding.
The system prompt anchors activation space the same way training data anchors weight space: through statistical pressure on what comes next.
The patterns repeat across layers of the system. Training, architecture, inference, usage. Layers within layers across densely packed weights in the network.
This is not a deep insight. but once we see the machinery, the mystique fades. The model isn’t doing something magical when it writes good code or drifts into sycophancy.
It’s doing math on probability distributions. Understanding that makes us better at using them.
“When you hit enter”, you are querying a frozen snapshot. The model cannot learn from your prompt. Even if you use RAG or an agent to inject additional context, you are only modifying the input state, the model itself remains static, routing those new tokens through the exact same frozen circuitry.
This is why the biggest lever for making a model smarter is packing more high-signal data into the weights before the freeze. And that single fact is driving the entire AI economy we have today in 2025-2026. It’s why AI labs are scraping every corner of the internet, triggering massive copyright lawsuits from publishers and artists.
The more impactful issue today is the violently expensive infrastructure required to store and process it all. To build and run these frozen matrices, High Bandwidth Memory (HBM) for AI accelerators is currently eating the global supply of DRAM wafers. which is why a standard DDR5 kit costs roughly twice what it did a year ago.
Well, if you got this far, thanks for reading and I hope this helped, until next time!!!!
References and Further Reading
