Agentic Eval

Agentic Eval is a GitHub Awesome Copilot skill for evaluating and improving AI-agent outputs with repeatable criteria rather than subjective impressions. It supports self-critique, evaluator-optimizer loops, rubric-based scoring, test-driven refinement and LLM-as-judge patterns.

• Teams building AI agents that need quality gates
• Developers testing prompt or agent changes
• Projects creating repeatable evaluation rubrics
• AI product teams comparing output quality across iterations

• Create a rubric for an agent task
• Run an evaluator-optimizer improvement loop
• Use LLM-as-judge with explicit scoring criteria
• Turn known failure cases into regression evaluations

The skill defines the desired behavior and evaluation criteria, builds representative cases, applies one or more evaluator patterns, records where outputs fail the rubric and iterates until the system meets the required threshold. It emphasizes explicit criteria so improvements can be measured rather than merely described.

• Makes agent quality measurable
• Supports several complementary evaluation patterns
• Turns failures into reusable regression cases
• Useful for continuous improvement of prompts and agents

• LLM judges can be biased or inconsistent
• Evaluation sets need representative tasks to be meaningful
• Human review is still important for subjective or high-impact criteria