Observing AI agents: traces, replays and cost per task

Picture a checkout task that should take about 12 steps. The trace shows 60. Most of them are the same cycle: screenshot, click “Continue”, screenshot, button still disabled, click again. The model never gave up because nothing told it to, and nobody noticed until the invoice.

Agents fail differently from ordinary software. They rarely crash. They keep going, confidently, in the wrong direction. So the question you have to answer fast is “what did it do, step by step, and what did that cost?”, and a normal application log can’t answer it.

This is how I’d set up observability for browser, computer-use and phone-use agents, from the minimum you need on day one to the extras that pay off later.

What you need

• A task ID that follows everything. Every model call, action, screenshot, log line and cost row carries it.
• Somewhere to send traces: OpenTelemetry into whatever backend you already run, or a tracing tool built for LLM apps. The data model matters more than the vendor.
• Cheap storage for screenshots and recordings, with a retention policy from day one.
• The token usage numbers your model API returns on every response.

Step by step

1. Model each task as a trace

One task is one trace. Inside it, one span per agent step. Inside each step, child spans for the model call and for each browser or device action.

OpenTelemetry’s GenAI semantic conventions give you names so you don’t invent your own: gen_ai.operation.name with values like chat, invoke_agent and execute_tool, plus gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. As of September 2026 they’re marked Development and live in their own repository, so expect renames and pin the version you build against.

On each step span, record:

• the step number, and the action the model chose with its target (selector, element index or coordinates)
• whether the post-condition passed
• a pointer to the screenshot, not the image itself
• the current URL, or the foreground app on a phone
• model latency and action latency as separate numbers

from opentelemetry import trace

tracer = trace.get_tracer("agent")

with tracer.start_as_current_span("task", attributes={"task.id": task_id}):
    for i in range(MAX_STEPS):
        with tracer.start_as_current_span("step") as span:
            span.set_attribute("step.index", i)
            resp = call_model(history)
            span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
            # run the chosen action, record target and post-condition

If it breaks: spans missing from the end of long tasks usually means the exporter’s batch was dropped when the worker got killed. Flush the exporter on shutdown and on task timeout.

2. Keep a screenshot for every step

For any agent that looks at a screen, the screenshot is the most useful single artefact you have. It’s what the model saw when it made the decision. Store it keyed by task and step, downscaled, and link it from the span.

Browser agents on Playwright get more for free. Playwright tracing captures DOM snapshots before and after each action, a screencast, network requests and console output in one zip you open with npx playwright show-trace. The docs note that the hosted viewer at trace.playwright.dev loads the trace in your browser without sending it anywhere, which matters once traces contain customer data. Recording everything is heavy. Playwright’s test runner has a retain-on-failure mode that keeps only failed runs, and it’s a sensible policy to copy in your own runner.

Phone agents should pair each screenshot with the UI tree dump from the same step. When the agent tapped the wrong thing, the tree tells you whether the element was mislabelled or the model misread it. Phone-use agents on real Android covers that observe loop in detail.

If it breaks: black screenshots from a phone usually mean the app marked that window as secure. That’s the app’s choice, so log it as a known blind spot and don’t retry.

3. Write step logs a person can read

Traces are for tools. You also want a plain-text log per task that someone can read top to bottom in two minutes:

task 8f3c  identity=acct-42  goal="download the March invoice"
step 1  goto /billing                  ok    1.8s
step 2  click "Invoices"               ok    0.9s
step 3  click "March 2026"             FAIL  no download started
step 4  click "March 2026"             FAIL  no download started
step 5  click "March 2026"             FAIL  no download started
stopped: same action on same target 3 times

If your agent states a reason for each action, log it next to the action. It’s the fastest way to spot a misunderstanding.

If it breaks: logs that nobody reads are usually too long. One line per step, with details behind the task ID.

4. Add replays where they earn their keep

A replay is the screenshots or a recording stitched into something you can scrub through. For Playwright agents the trace viewer already is one. For phones, adb shell screenrecord records the screen to an MP4, with options for size, bit rate and a time limit.

Nobody watches replays routinely. They’re for three situations: a customer asks what happened on their task, a new failure the log can’t explain, and reviewing a sample of successful runs to find the ones that “succeeded” wrongly. That third one is where the surprises are.

If it breaks: replays eat storage quickly. Keep them for failures and a sample of successes, and expire the rest.

5. Track cost per task

Every model API response reports token usage. Multiply by your price for that model, add it to the task, and store the total next to the outcome. Per task, track:

• model cost, split into input and output tokens
• step count and the number of screenshots sent
• browser or device minutes, and proxy bandwidth if you pay per GB
• human minutes, if a checkpoint was involved
• the outcome: succeeded, failed, stopped by a limit, or handed to a human

Then report cost per successful task: total spend divided by successes. Failed tasks still cost money, so this is the real unit cost. It moves when the success rate moves, even if nothing about per-token pricing changed, and it’s the number that decides whether an agent is worth running at all.

Screenshots pile up fast in computer-use loops. Anthropic’s computer use docs recommend resizing screenshots and pruning older ones in long runs. Track screenshots per task as its own metric, because it’s the first thing to creep up.

If it breaks: cost that doesn’t reconcile with the provider’s invoice usually means retries or failed calls aren’t being attributed to a task. Record usage at the lowest level, the individual call, and roll up.

6. Alert on loops and drift

Things worth alerting on, roughly in the order they’ll save you money:

• the same action on the same target three times in a row
• steps per task above some multiple of that task type’s median
• cost per task over a hard ceiling (the task should already have stopped itself; the alert tells you the stop didn’t fire)
• the success rate for one task type dropping over an hour
• a jump in challenge pages, 403s or 429s from one domain, which is almost always an access problem and not a model problem; why AI browser agents get blocked covers the causes
• more idle-but-unhealthy browsers or devices in the pool than usual

The first three should also be hard stops inside the agent. Alerts are for when your stops have bugs.

If it breaks: an alert that fires daily gets ignored within a week. Tune thresholds per task type, not globally.

Common pitfalls

• Storing full prompts and screenshots with no redaction. Your traces quietly become the most sensitive data store you own. Redact credentials and personal data before storage, and set retention.
• Measuring cost per call. It hides retries and failures.
• Trusting the agent’s own “done”. Check the outcome independently: the file exists, the order is in the list, the message is in the sent folder.
• Never reviewing successes. Some of them are wrong, and only a person looking at replays will find those.

Scaling this

With a few dozen tasks a day, one person can read every step log. Once it’s thousands, move to dashboards per task type (success rate, median steps, cost per success) and a small daily sample of replays. Trace storage becomes a real line item at that point, so sampling successful traces while keeping every failed one is the usual compromise.

Where to go next

The limits, caps and human checkpoints that pair with these alerts are in the browser agent production checklist. Everything else is in the article index.

I haven’t tested every LLM tracing product on the market, so this piece sticks to the data model and to OpenTelemetry, which most of them can ingest.

Written by Xavier Fok

disclosure: The Agent Ops Report is published by the team that runs Singapore Mobile Proxy and cloudf.one. Any link to those two products is ours and is marked sponsored. No vendor paid for a mention in this article. Last reviewed by Xavier Fok on 2026-09-11.