# Agent Observability Specification Builder

Specify the traces, metrics, logs, alerts, and review views needed to understand an agent's decisions and diagnose failures.

## Prompt

You are an AI reliability engineer who specializes in observability for production agents.

Inputs:
1. Agent workflow and business objective: {{agent_workflow}}
2. Models, tools, data sources, and infrastructure: {{system_components}}
3. Success targets and service-level objectives: {{success_targets}}
4. Known incidents and difficult-to-diagnose failures: {{failure_history}}
5. Privacy, security, retention, and tooling constraints: {{constraints}}

Do the following:
1. Map the end-to-end execution path and define a trace structure for user request, planning, model calls, retrieval, tool calls, approvals, retries, final outcome, and downstream effects.
2. Choose outcome, quality, reliability, latency, token, cost, safety, and human-intervention metrics; define each metric's formula, dimensions, and owner.
3. Specify structured event schemas and correlation IDs while redacting secrets, personal data, hidden reasoning, and unnecessary content.
4. Create alert conditions based on user impact and sustained deviation, with diagnostic context, severity, routing, and runbook entry for each alert.
5. Produce an observability matrix, dashboard layout, sample sanitized events, alert catalog, retention policy, and three incident investigation walkthroughs.

## Best for

AI platform and reliability teams that need actionable production visibility without logging sensitive agent content indiscriminately.

## Compatible tools

- Claude
- ChatGPT

## How to use

- Map every model, retrieval, tool, and human-review stage.
- Provide numerical success and service-level targets.
- List sensitive fields that must be redacted or omitted.
- Connect each alert to a named owner and runbook.

## Customization tips

- Measure user outcomes instead of only model latency.
- Tag traces with prompt, model, and policy versions.
- Separate expected denials from system errors.
- Retain aggregate metrics longer than sensitive event content.

## Example input

Workflow: An IT service-desk agent diagnoses employee issues, searches internal knowledge, runs approved device checks, and opens or updates tickets. Components: two language models, hybrid search, endpoint-management API, identity directory, and ticketing system. Targets: 75% containment, p95 response under 12 seconds, tool-error rate under 2%, and zero cross-user data exposure. Incidents: repeated tool retries caused duplicate tickets; stale articles led to incorrect VPN steps. Constraints: do not log employee messages beyond 14 days, never store access tokens, and use existing OpenTelemetry infrastructure.

## Example output

The trace links request, retrieval, model, tool, and ticket spans with a session-safe correlation ID and idempotency key. Core metrics include verified containment rate, duplicate-ticket rate, unsupported-resolution rate, retrieval freshness, p95 latency, cost per resolved request, and approval frequency. Sanitized events store article IDs and tool status but redact message bodies after 14 days and never capture tokens. Alerts fire on sustained duplicate creation, stale-source usage, authorization denials, and containment drops, each with owner and runbook steps.