Forget the marketing. When someone says "we're adding AI to our product," they usually mean they are making API calls to a large language model. That decision deserves a clearer mental model than "it's smart autocomplete." What follows is a practical breakdown of how LLMs work — written for engineers and product teams who need to reason about these systems, not just consume them.
Tokens Are the Atomic Unit, Not Words
The first thing to understand is that LLMs do not read words. They read tokens — chunks of text that roughly correspond to word fragments, punctuation, or whole short words depending on the tokenizer used. "Unbelievable" might become three tokens. A space before a word is often its own token.
This matters in practice because:
- Pricing on most LLM APIs is per token, not per word or per request.
- Context windows (how much the model "remembers" in one conversation) are measured in tokens.
- Seemingly simple prompts can be surprisingly expensive at scale.
A 4,000-token context window sounds large until you realize a moderate-length document and its system prompt can fill it quickly.
The Transformer Architecture: Attention Is Everything
Modern LLMs are built on the Transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need." The core insight was replacing recurrent networks with a mechanism called self-attention, which allows the model to weigh the relevance of every token to every other token in the input simultaneously.
Think of it this way: when reading the sentence "The bank by the river flooded," a human instantly knows "bank" means a riverbank and not a financial institution because of surrounding context. Self-attention does something analogous — mathematically. Each token queries every other token and learns to assign different weights based on what is contextually relevant.
Stacked layers of these attention heads, combined with feed-forward neural networks, allow the model to build increasingly abstract representations of language — from grammar to facts to reasoning patterns.
Input Text → Tokenizer → Embedding Layer
→ [Transformer Block × N]
- Multi-Head Self-Attention
- Layer Normalization
- Feed-Forward Network
→ Output Logits → Softmax → Next Token Prediction
Pre-training: Learning the Shape of Language
LLMs are first trained on massive corpora of text — books, websites, code, research papers — using a deceptively simple objective: predict the next token. This is called self-supervised pre-training because the labels are already in the data itself; no human annotation is required at this stage.
Through billions of gradient updates across trillions of tokens, the model does not just memorize text. It compresses statistical patterns about how language, facts, and reasoning are structured. The result is a base model with broad capability but no particular behavior or personality.
Fine-tuning and RLHF: Teaching the Model to Be Useful
A raw pre-trained model is not very usable. It will happily complete a prompt in a direction you did not intend. This is where fine-tuning comes in — specifically a technique called Reinforcement Learning from Human Feedback (RLHF).
The process works in three rough stages:
- Supervised fine-tuning (SFT): Human labelers write ideal responses to prompts. The model is trained to imitate these.
- Reward modeling: A separate model is trained to score responses based on human preferences.
- RL optimization: The LLM is updated using the reward model's scores to reinforce helpful, harmless, and honest behavior.
This is how a chaotic next-token predictor becomes a structured assistant that follows instructions, refuses dangerous requests, and maintains a tone. It also explains why different models from the same base (GPT-4, Llama 3, Mistral) behave so differently — fine-tuning choices shape nearly everything the end user experiences.
Temperature, Sampling, and Why the Same Prompt Returns Different Answers
At inference time, the model outputs a probability distribution over all possible next tokens. Temperature controls how "peaked" or "flat" that distribution is:
- Low temperature (e.g., 0.1): The model almost always picks the highest-probability token. Output is deterministic and conservative.
- High temperature (e.g., 1.2): Lower-probability tokens get more of a chance. Output is more creative, but also more likely to hallucinate.
Other sampling strategies like top-k and top-p (nucleus sampling) further constrain which tokens are even in the running before temperature is applied. Understanding these parameters is critical when building LLM-backed features — defaulting to whatever the API provides is rarely the right call for production systems.
What LLMs Cannot Do (By Design)
This is the part that matters most for engineering decisions.
LLMs have no persistent memory across sessions unless you build it. They have no access to real-time information unless you provide it via retrieval (RAG). They do not reason in the way a logic engine does — they pattern-match at enormous scale, which can look like reasoning but breaks in systematic ways. They can be confidently wrong.
These are not bugs waiting to be fixed. They are architectural properties of how these systems are built. Treating them otherwise leads to products that fail users in predictable ways.
This is why serious LLM applications today are not just raw API calls. They are systems: retrieval pipelines, structured output parsers, guardrails, evaluation loops, and careful prompt engineering layered on top of the model itself.
Why This Matters for Your Project
If you are building a SaaS product, a mobile application, or an internal tool with any LLM component, understanding this architecture is the difference between shipping a reliable feature and shipping a liability. Knowing that temperature affects reliability, that context windows have hard limits, that RLHF shapes behavior, and that hallucination is probabilistic rather than random — all of it leads to better architecture decisions, more honest stakeholder conversations, and faster debugging when something goes wrong. The model is one layer in your stack. Design accordingly.
Source: "How LLMs Actually Work" — https://www.0xkato.xyz/how-llms-actually-work/ via Hacker News
