From token counts to business outcomes: the next era of AI observability

The first generation of AI observability helped teams answer a narrow but important question:

Did the model call work?

That was the right starting point.

When companies first started putting large language models into applications, they needed visibility into prompts, completions, latency, token usage, errors, and traces. They needed to debug broken chains, inspect bad responses, and understand why a model call behaved differently than expected.

But AI systems are changing.

The unit of work is no longer just a single model call. AI agents now call tools, make decisions, run workflows, trigger actions, access files, retry steps, and operate across both internal and customer-facing environments.

That means AI observability has to evolve too.

The next generation of AI observability will not stop at token counts. It will connect AI activity to cost, ownership, product context, risk, and business outcomes.

The first era: watching the model call

Early AI observability focused on the model interaction.

That usually meant tracking:

• prompt
• completion
• model
• token count
• latency
• error rate
• trace
• cost estimate

These signals are still useful. Teams need to know whether a model call succeeded, how long it took, and how much it cost.

But these signals mostly describe consumption.

They tell you that tokens were used. They do not tell you whether the work was valuable.

A trace may show that a customer reply was generated. It may not show whether the user accepted it, edited it, sent it, or discarded it.

A provider invoice may show that a model cost increased. It may not show whether the increase came from a successful feature, a few high-usage customers, a model routing bug, or a runaway workflow.

A latency chart may show that a request took three seconds. It may not show whether that delay mattered to the user experience.

The model call is important, but it is not the whole story.

Why token counts are not enough

Token counts are easy to measure, which makes them tempting to overuse.

But tokens are not a business metric.

A cheap model call can still be waste. An expensive model call can still be worth it. The difference depends on what the AI work produced.

Consider three examples:

• A support agent spends $0.03 generating a response that is accepted and sent by a human reviewer.
• A sequence generator spends $0.003 generating drafts that users discard 90% of the time.
• A coding agent runs unattended for hours and burns through API calls because a loop never exited.

The token count alone does not tell you which of these is healthy.

To understand AI work, teams need to know what happened before and after the model call:

• Who triggered it?
• Which agent or workflow ran?
• Which product feature or internal project did it belong to?
• Which customer or team created the usage?
• Which model was selected?
• Was the output accepted, edited, sent, ignored, or rejected?
• Did the workflow complete?
• Did the session behave normally?

Without that context, AI cost is just a line item.

With that context, AI cost becomes an operating signal.

The shift to AI work attribution

The next era of AI observability is about attribution.

Not just "what model was called," but:

What AI work happened, who caused it, what did it cost, and what outcome did it produce?

That means moving from model-level telemetry to session-level telemetry.

A useful AI session record should include:

• agent name
• user or service that started the run
• session ID
• model usage
• tool calls
• latency
• token usage
• cost
• risk signals
• product or project context
• outcome

For internal AI tools, context might include the developer, repository, project path, coding agent, model, and local workflow.

For product AI features, context might include the customer, account plan, product screen, user action, entity ID, workflow ID, and whether the output was accepted.

This is the difference between observing AI calls and understanding AI work.

The metric that matters: cost per useful outcome

One of the clearest examples of this shift is the move from cost per token to cost per useful outcome.

Cost per token tells you how much consumption occurred.

Cost per useful outcome tells you whether that consumption produced value.

A useful outcome might be:

• a generated reply accepted by a user
• a sequence sent
• a document classification completed
• a support ticket resolved
• a summary used
• a workflow completed without human rework

This metric changes the conversation.

Instead of asking, "Why did our AI bill go up?" a product or engineering leader can ask:

• Which features are producing the most useful AI outcomes?
• Which customers are expensive but valuable?
• Which workflows are costly and low-value?
• Which models produce the best outcome per dollar?
• Which outputs are repeatedly discarded?
• Which parts of the product are leaking margin?

That is a much better operating conversation.

Why this matters across the company

AI observability is no longer only a developer debugging problem.

It affects multiple teams.

Engineering needs to know which agents are running, how they behave, and whether they are reliable.

Product needs to know which AI features are being used, which ones produce value, and which ones hurt margin.

Finance needs to understand where AI spend is going and how to attribute it.

Security and governance teams need to know what agents did, which tools they used, what data they touched, and whether approvals were required.

Executives need a clear summary of AI cost, adoption, risk, and business impact.

That requires a shared operating layer, not five disconnected dashboards.

Local agents change the surface area

One reason this problem is becoming urgent is that AI agents are not only running in production.

They are also running locally.

Developers use tools like Claude Code, Cursor, Codex, Cline, Windsurf, and local model runners as part of daily work. These agents may interact with proprietary repositories, local files, terminals, and internal systems.

That creates a new visibility gap.

Most companies can see production logs. Far fewer can understand local AI agent activity across engineering teams.

But local agent usage still creates operational questions:

• Which tools are being used?
• Which repositories are involved?
• Which sessions ran unusually long?
• Which models are driving spend?
• Which workflows happened after hours?
• Did an agent run unattended?
• Which local activities map to real cost?

If agents are becoming part of the work, teams need visibility into where that work happens.

What the next AI observability stack needs

The next AI observability stack should still include traces. Traces are useful.

But traces are only one layer.

Teams will also need:

• session-level action ledgers
• cost attribution by model, agent, user, team, project, feature, and customer
• product context for AI workflows
• local agent visibility
• model routing analysis
• anomaly detection
• outcome tracking
• approval and policy checkpoints
• evidence exports
• dashboards for engineering, product, finance, and governance

The goal is not to create more noise.

The goal is to give teams a trustworthy view of AI work across the company.

The end of token-only thinking

Token counts were a useful starting point.

They gave teams a way to measure usage when AI systems were mostly model calls. But as agents become more capable, more distributed, and more embedded in real workflows, token counts alone will not be enough.

The real question is not just how many tokens were used.

The real question is whether the AI work was worth it.

That is where AI observability is heading: from traces to sessions, from costs to attribution, and from token counts to business outcomes.