LLMs Are Their Weights: What That Really Means for AI Software
Ask a software engineer what a program is, and they will point to source code — readable logic that can be audited, debugged, and version-controlled line by line. Ask the same question about a large language model, and the honest answer is: a very large file of numbers.
That distinction is not academic. It has real consequences for every team building products on top of AI today.
What "Made of Weights" Actually Means
A trained neural network encodes everything it knows — grammar, facts, reasoning patterns, coding conventions, cultural context — inside its parameters. These are typically 32-bit or 16-bit floating-point numbers, billions of them, arranged in matrices that transform input tokens into output tokens through successive layers of multiplication and addition.
There is no lookup table. There is no hand-written rule that says "a capital city is the seat of government." That relationship is implied by the distribution of weight values learned during training on hundreds of billions of tokens of text.
This is fundamentally different from every other class of software artifact:
- A relational database stores facts explicitly. You can query, update, or delete them.
- A rule engine encodes logic explicitly. You can read the rules and trace decisions.
- A trained model stores neither facts nor logic explicitly. It stores tendencies, compressed and entangled across every parameter.
Why This Creates Unique Engineering Challenges
Opacity Is Structural, Not Accidental
When a model gives a wrong answer, there is no stack trace to follow. The "bug" is not a misplaced conditional — it is a statistical bias embedded somewhere across billions of weights. You cannot grep for it. This is why interpretability research exists as its own field, and why production AI systems need robust evaluation pipelines that treat the model as a black box even when you built it yourself.
Knowledge Has a Cutoff — and It Decays
Weights are frozen at the end of training. The model does not learn from user interactions at inference time (unless you explicitly fine-tune). This means every deployed model is, in a sense, a snapshot of the world at a particular moment. For SaaS products that surface time-sensitive information, this is a critical architectural constraint — and one that drives the adoption of retrieval-augmented generation (RAG) as a patch layer on top of static weights.
Size Is a First-Class Deployment Concern
A 7-billion-parameter model at 16-bit precision occupies roughly 14 GB of memory. A 70-billion-parameter model needs approximately 140 GB — more than most cloud VMs carry in GPU VRAM by default. Shipping an LLM is not like deploying a new microservice. Infrastructure decisions (quantization level, model sharding, batching strategy) are inseparable from the product decisions. Founders who discover this late pay for it in cloud bills and latency.
The Weights Are the Product
Here is the strategic implication that most product teams underestimate: for an AI-native company, the weights are the moat — or they are not, and you need to be honest about that.
If you are calling a third-party API like OpenAI or Gemini, you are renting access to someone else's weights. Your product differentiation must live elsewhere: in your prompting strategy, your retrieval layer, your fine-tuning pipeline, your UX, your domain-specific data flywheel. That is not a weakness — many excellent products are built this way — but it is a structural reality you should design around intentionally.
If you are training or fine-tuning your own weights, you are making a significant bet on compute, data quality, and ongoing evaluation infrastructure. The barrier to entry is real, but so is the defensibility.
# Rough memory estimate for a model in GB
def estimate_model_memory_gb(param_count_billions: float, bits: int = 16) -> float:
bytes_per_param = bits / 8
total_bytes = param_count_billions * 1e9 * bytes_per_param
return total_bytes / (1024 ** 3)
# Examples
print(estimate_model_memory_gb(7)) # ~13.0 GB
print(estimate_model_memory_gb(70)) # ~130.4 GB
print(estimate_model_memory_gb(405)) # ~754.6 GB
Practical Takeaways for Software Teams
1. Treat the model as infrastructure, not logic. Your business rules should not live inside a prompt. They belong in your application layer where they can be tested, versioned, and changed without retraining anything.
2. Build evaluation before you build features. Because you cannot read weights like code, the only way to know if your model is behaving correctly is to measure its outputs systematically. An eval suite is not optional — it is your test coverage.
3. Plan for model swaps. Weights get replaced. GPT-4 becomes GPT-4o becomes whatever comes next. Architect your integration so that switching the underlying model is a configuration change, not a refactor.
4. Data is the input to weights. If you want a model that understands your domain, the path runs through data curation and fine-tuning — not prompt engineering alone. Invest in your data pipeline early.
A Different Mental Model for a Different Technology
The software industry spent decades building intuitions around deterministic, inspectable systems. LLMs violate nearly every one of those intuitions. They are probabilistic, opaque, and stateful in a way that cannot be queried — only probed.
Recognising that a model is its weights is the first step toward reasoning clearly about what it can and cannot do, where to trust it, and where to build guardrails around it.
Source: Max Leiter, "They're made out of weights" — https://maxleiter.com/blog/weights
Why this matters for your project: Whether you are integrating a hosted LLM into a SaaS product or training a custom model for a domain-specific application, understanding the weight-centric nature of these systems shapes every decision from infrastructure sizing to data strategy. At Code!nk Technologies, this is the lens we apply when helping clients move from AI prototype to production — because the gap between the two is almost always about architecture, not model quality.
