Tokenomics in Agentic AI: Where Do All the Tokens Actually Go?
Every API call to a large language model costs money. That much every SaaS founder knows. But when you move from a simple chat completion to a full agentic system — one that plans tasks, calls tools, reads files, and iterates on code — the token bill becomes something else entirely. Recent research has started putting hard numbers on this, and the results should change how engineering teams design and budget for agentic software.
What "Agentic" Actually Means in Practice
An agentic AI system is not a single prompt-response pair. It is a loop. A typical software engineering agent might:
- Receive a high-level task ("Fix the failing test in
auth_service.py") - Plan a sequence of steps
- Call tools — file readers, shell executors, search utilities
- Receive tool outputs and reason over them
- Generate code or edits
- Verify the result and iterate
Each one of those steps consumes tokens — both as input (the context the model reads) and output (what it writes back). The cumulative cost is not linear with task complexity. It compounds.
Breaking Down the Token Budget
Research into agentic software engineering workflows reveals that token consumption clusters into three major categories, and the distribution is often counterintuitive.
1. Context and System Prompts
The agent's instructions, tool schemas, and memory state are re-injected on every single model call. In a system with ten tools defined, a full conversation history, and a working scratchpad, the system prompt alone can consume thousands of tokens — per turn. Over a multi-step task, this is frequently the largest single cost center, not the actual code generation.
2. Tool Call Inputs and Outputs
When an agent reads a file, it gets the file back as tokens. When it searches a codebase, the results return as tokens. These "environmental observations" are easy to overlook in architecture discussions but can dwarf the cost of the agent's own reasoning. A single tool call that returns a 500-line file is burning context window space that cascades through every subsequent step.
3. Chain-of-Thought and Reasoning Traces
Many capable agents are prompted — or model-intrinsically inclined — to reason before acting. This inner monologue is valuable for accuracy, but it is also output tokens, which are typically priced higher than input tokens by most providers. A verbose reasoning trace is a direct cost multiplier.
Here is a rough illustration of how token consumption can look across a single multi-step task:
Task: "Refactor the payment module to support multi-currency"
Step 1 — Plan generation:
Input: ~2,400 tokens (system prompt + task)
Output: ~600 tokens (plan)
Step 2 — File reads (3 files via tool):
Input: ~5,200 tokens (system prompt + history + tool results)
Output: ~300 tokens (next action)
Step 3 — Code generation:
Input: ~6,800 tokens (full context now)
Output: ~1,100 tokens (generated code)
Step 4 — Verification loop (2 iterations):
Input: ~9,400 tokens each
Output: ~400 tokens each
Total: ~32,000+ tokens for one feature-level task
That is a real number for a moderately complex task. Scale that to a CI pipeline running dozens of agents per day and the economics demand engineering attention.
Why This Should Change How You Architect Agents
Understanding token distribution leads directly to architectural decisions that reduce cost without sacrificing capability.
Trim tool output aggressively. Do not return raw file contents when a summary or a targeted excerpt will do. A retrieval layer that returns only the relevant function rather than the full module can cut input tokens by 60–80% per observation step.
Compress conversation history. Rolling summaries or sliding window truncation prevent the context from growing unboundedly. Most agents do not need the verbatim record of every prior step — they need the salient state.
Separate planning from execution models. A smaller, cheaper model can handle planning and routing. Reserve the expensive frontier model for actual code synthesis. This kind of model tiering is now straightforward with most orchestration frameworks.
Cache tool schemas and system prompts. Providers like Anthropic and OpenAI offer prompt caching that can dramatically reduce the cost of re-sending large, static system prompts on every turn. This alone can cut total spend by 20–40% on prompt-heavy agents.
Budget per task, not per call. Once you know your approximate token distribution, you can set hard budgets per agent task and implement early-exit logic when the budget is nearly exhausted. This prevents runaway loops from becoming runaway invoices.
The Measurement Problem
One underappreciated finding in this research area is that most teams are flying blind. Token usage is logged at the API level, but few engineering teams attribute that usage back to specific agent steps, tools, or task types. Without that granularity, optimisation is guesswork.
Instrumenting your agent to log tokens consumed at each step — input, output, and which tool triggered the call — is table stakes before any meaningful cost optimisation. Libraries like LangSmith, Langfuse, and Helicone make this relatively straightforward to wire in without changing core agent logic.
Implications for SaaS Products Built on Agents
If you are building a SaaS product where agentic AI is a core feature — a coding assistant, an autonomous data analyst, a document processing pipeline — your token economics are effectively your cost of goods. Underestimating this leads to unit economics that look fine in a demo environment and catastrophic at scale.
The right framing is to treat token budget design the same way you treat database query optimisation: measure first, identify the expensive operations, and apply targeted fixes. The gains are real. Teams that have gone through this process systematically report 40–70% reductions in per-task token spend without meaningful degradation in output quality.
Source: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering — https://arxiv.org/abs/2601.14470, via Hacker News.
Why this matters for your project: Whether you are shipping an internal AI dev tool or embedding agents into a customer-facing product, token cost is now an engineering discipline of its own. At Code!nk Technologies, we factor token budgeting into the architecture of every AI-powered system we build — because a product that works in staging but bleeds money in production is not a finished product.
