# Agent Evaluation Benchmark Builder

Create a representative benchmark that tests an AI agent's task success, tool use, safety, recovery, and efficiency before release.

## Prompt

You are an AI evaluation lead who specializes in benchmark design for tool-using agents.

Inputs:
1. Agent purpose and target users: {{agent_context}}
2. Tools, permissions, and environment: {{tools_and_environment}}
3. Common tasks and critical edge cases: {{task_distribution}}
4. Known failure modes and unacceptable outcomes: {{failure_modes}}
5. Release criteria, budget, and test resources: {{evaluation_constraints}}

Do the following:
1. Define the evaluation dimensions, including task completion, answer quality, tool selection, argument accuracy, safety, recovery, latency, and cost; assign justified weights.
2. Build a test-set blueprint that reflects real task frequency while reserving enough cases for rare high-impact failures, adversarial inputs, and multi-step recovery.
3. Specify each test case's initial state, user request, hidden ground truth, allowed actions, expected trajectory, pass conditions, and automatic versus human scoring method.
4. Design repeat-run and regression procedures that account for nondeterminism, environment changes, evaluator disagreement, and benchmark contamination.
5. Produce a scorecard, at least 12 example test cases, release gates, failure-severity rules, and instructions for expanding the benchmark from production incidents.

## Best for

AI engineering and quality teams that need defensible release gates for agents performing multi-step work with tools.

## Compatible tools

- Claude
- ChatGPT

## How to use

- Supply production-like task frequencies and edge cases.
- Name every tool permission and prohibited action.
- Define measurable release thresholds before generating tests.
- Run critical cases multiple times to expose nondeterministic failures.

## Customization tips

- Add anonymized incidents as regression cases.
- Use separate thresholds for critical and routine tasks.
- Specify whether graders can inspect the agent trajectory.
- Reserve a hidden test set to reduce benchmark gaming.

## Example input

Agent context: A recruiting coordinator agent for a 300-person software company. Tools: ATS read/write, calendar search, email drafting, and interview-kit retrieval; it cannot send emails without approval. Task mix: 55% scheduling, 25% candidate status questions, 15% interviewer preparation, 5% rescheduling conflicts. Failure modes: exposing another candidate's data, booking outside working hours, wrong time zones, or advancing a candidate without authorization. Release criteria: zero critical privacy failures, at least 92% scheduling success, and median cost below EUR 0.18 per task.

## Example output

The benchmark weights task success at 30%, privacy and authorization at 30%, tool correctness at 15%, recovery at 10%, communication quality at 10%, and efficiency at 5%. Its 120-case blueprint mirrors normal traffic but includes 20 dedicated privacy and authorization cases. Example tests cover DST transitions, two candidates with similar names, revoked calendar access, unavailable interviewers, malformed ATS data, and an attempted unauthorized stage change. Release requires zero severity-1 failures across three runs, 92% scheduling success, and the stated cost limit.