A few weeks ago my research agent started costing four times what it had the week before. Same feature, same traffic, same model. The output looked fine. The bill did not.
I spent an afternoon adding print statements before I admitted the obvious: I had no idea what the agent was actually doing on any given run. It made a variable number of tool calls, retried on failures I never logged, and occasionally fell into a loop where it re-read the same document three times before giving up. None of that showed up anywhere except the invoice.
The culprit turned out to be a retry path. A flaky tool was timing out, the agent was silently retrying with the full context re-sent each time, and a single "successful" run was making nine model calls instead of two. I only found it because I finally wrote down what every step was doing. That afternoon cost me more than a month of any tracing tool would have.
The short version: an agent is a loop you don't fully control, and you cannot improve a loop you can't watch. You need to capture, per run, what happened at each step, how many tokens it burned, what it cost, how long it took, and whether it failed. You do not need a platform to start. You need a log.
What to actually capture
Forget the tooling for a second. The question is: when a run goes wrong, what would you want to know? Work backwards from that and you get the list.
For every agent run, I want one trace — the whole run — made of spans, one per step. Each span should carry:
- What kind of step it was — a model call, a tool call, a retrieval, the top-level agent invocation.
- Token usage — input tokens and output tokens, separately. They cost different amounts and they tell you different things.
- Cost — derived from tokens and the model's price. Compute it once, store it on the span.
- Latency — start and end time. Slow runs and expensive runs are usually different problems.
- Tool calls — which tool, what arguments, what it returned (or how it failed).
- Outcome — did the step succeed, fail, or retry, and why.
- Eval score — if you score the run (you should), attach the score so you can correlate quality with cost.
That last point is the one people skip, and it's the one that pays off. A trace tells you what happened; an eval score tells you whether it mattered. Joining them lets you ask the only question that counts: is the expensive version actually better? Often it isn't.
Notice what's not on the list: full prompt and response text on every span by default. Capturing message content is useful for debugging and risky for privacy and storage cost. I capture it behind a flag I can turn on for a specific run, not always-on for everything.
OpenTelemetry's GenAI conventions, as a baseline
Before you invent your own field names, look at what already exists. OpenTelemetry — the same vendor-neutral standard you'd use for any backend tracing — has a set of GenAI semantic conventions: an agreed vocabulary for naming the things I just listed.
The useful part is the attribute names. A model-call span gets gen_ai.request.model (which model), gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (the token split), gen_ai.response.finish_reasons (why it stopped — stop, tool_calls, and so on), and gen_ai.provider.name to mark the flavor. There are even attributes for prompt-cache accounting — gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens — which matter a lot once caching is in play, because cached input can be an order of magnitude cheaper. Span names follow the format {gen_ai.operation.name} {gen_ai.request.model}, and the agent-level conventions define operations like invoke_agent (the whole run), chat (a single model call), and execute_tool (a tool invocation). (OTel GenAI spans, OTel GenAI observability blog)
Two honest caveats. First, as of mid-2026 these conventions are still marked Development status — they're stabilizing but not frozen, so attribute names can still shift. They recently moved to their own repository, which tells you they're being taken seriously and that they're still in motion. Second, you don't have to adopt OTel machinery to benefit. Even if you write a plain JSON log, use these field names. The day you outgrow your homemade log and move to a real backend, your data already speaks the language. It's the cheapest forward-compatibility you'll ever buy.
The homemade JSONL log that gets you 80% there
Here's the setup I actually run for small projects. One append-only JSONL file, one object per span, OTel-style field names. No service, no SDK, no account.
import json, time, uuid
PRICES = { # USD per token; see your provider's pricing page
"claude-haiku-4.5": {"in": 1.00/1e6, "out": 5.00/1e6},
"claude-sonnet-4.6": {"in": 3.00/1e6, "out": 15.00/1e6},
}
def log_span(trace_id, kind, model, usage, t0, status, **extra):
p = PRICES.get(model, {"in": 0, "out": 0})
cost = usage["in"] * p["in"] + usage["out"] * p["out"]
record = {
"trace_id": trace_id,
"span_id": uuid.uuid4().hex[:12],
"gen_ai.operation.name": kind, # invoke_agent | chat | execute_tool
"gen_ai.request.model": model,
"gen_ai.usage.input_tokens": usage["in"],
"gen_ai.usage.output_tokens": usage["out"],
"cost_usd": round(cost, 6),
"latency_ms": int((time.time() - t0) * 1000),
"status": status, # ok | error | retry
**extra, # feature, tool_name, eval_score...
}
with open("traces.jsonl", "a") as f:
f.write(json.dumps(record) + "\n")
Every step in your agent loop calls log_span with a shared trace_id. Tag each run with the feature it served (feature="research", feature="summarize"). Now you can answer real questions with a one-liner over the file:
# cost per feature, last run-set
jq -s 'group_by(.feature)[] | {feature: .[0].feature,
usd: (map(.cost_usd) | add)}' traces.jsonl
That's it. Per-run cost, per-feature attribution, latency, retry counts, and — if you write your eval score into the span — quality-versus-cost, all from a file you can read with jq, grep, or DuckDB. The retry loop that quadrupled my bill would have shown up here as status: "retry" rows stacked under one trace_id, three minutes of looking instead of an afternoon of guessing.
The price table is the one thing you must keep honest. Output tokens currently cost about 5x input across the current Claude tiers, and the headline rates I used above were $1/$5 per million for Haiku 4.5 and $3/$15 for Sonnet 4.6 when I checked — but rates change and caching/batch discounts move the real number a lot, so treat any table as a snapshot and check your provider's page. (Claude API pricing breakdown)
Token-cost attribution: the question that actually pays
Here's where the discipline earns its keep. Once every span carries a cost and a feature tag, you can stop guessing about your unit economics.
I ran this on my own project and learned that one "free" feature — an auto-summarize that fired on every document upload — was 60% of my model spend, used by maybe a tenth of my users. It wasn't a bug. It was a product decision I'd made blind. Seeing cost per feature turned it into a decision I could make on purpose: I made it opt-in, and the bill dropped by half with no complaints.
That's the whole point of attribution. Total spend is a number you can't act on. "Feature X costs $0.40 per run and feature Y costs $0.02" is a number you can. And when you join it to eval scores, you get the sharpest version: cost per good run. If the pricey path scores the same as the cheap one, the pricey path is just a leak with a nice description.
Build versus buy: when the JSONL log isn't enough
The homemade log is genuinely fine for a long time. It stops being fine when you want things a file can't give you cheaply: a UI to click through a single messy trace, automatic instrumentation so you're not hand-calling log_span everywhere, eval runs wired into CI, or sharing traces with someone who doesn't live in a terminal.
At that point the hosted tools are worth a look, and the honest summary is that they cluster by what they're best at, not by which is "best." A few I've used or evaluated:
- Langfuse captures the prompt, response, token usage, latency, and cost per trace, and is MIT-licensed and self-hostable — the natural next step from a JSONL file if you want a UI but want to keep your data. (It was acquired by ClickHouse in early 2026, which is worth knowing if vendor stability matters to you.)
- Arize Phoenix is OpenTelemetry-native — built on OTel and OpenInference instrumentation — and runs locally via Docker, with a strong open-source library of eval metrics. If you bought into the OTel conventions above, this is the lowest-friction backend.
- LangSmith is the deepest if you're already on LangChain/LangGraph, and less so if you're not.
- Braintrust leans eval-first, with CI gates that can block a merge when quality regresses — useful when shipping discipline matters more than dashboards.
I'm not going to crown one. The decision rule I use: if I want a dashboard, I look at Langfuse or Phoenix; if I want eval gates in CI, I look at Braintrust; if I'm deep in LangChain, LangSmith. And if I want neither yet, I stay on the JSONL file, because the file already answers the questions I have today. It's also worth noting the major coding agents — Claude Code, Codex, Copilot — now export OTel telemetry themselves, so some of this you get without writing any glue.
When this is overkill
Let me be the one to say it: for a brand-new prototype with three users and one feature, full tracing is procrastination dressed as rigor. If you can read every run by eye and your monthly bill is a rounding error, log the token counts and the cost, skip the rest, and go build the thing.
The honest threshold is two signals: when a bill surprises you, or when a failure hides from you. The first time you can't explain a cost spike, or the first time an agent fails in a way you can't reproduce, you've crossed the line — and the fix is an afternoon of adding log_span, not a week of standing up a platform.
Start with the file. Use the OTel field names so today's hack is tomorrow's migration path. Tag every span with a feature and a cost. Buy a tool the day the file stops answering your questions — not before. The goal was never observability for its own sake. It was being able to look at a run that went wrong and know, in three minutes, exactly where the money went.