Start with work that mattered
Synthetic evals are useful for coverage, but production traces show where the system actually meets the business. They capture messy inputs, partial context, tool latency, permissions, missing data, ambiguous instructions, and user behavior that no clean prompt set will fully anticipate.
The best source material is not only catastrophic failure. Escalations, manual corrections, long-running traces, abandoned sessions, near misses, and expert overrides all reveal places where the agent needs clearer constraints or better recovery.
- Customer escalations and support corrections.
- Agent runs that timed out, looped, or exceeded cost budgets.
- Cases where a human reviewer changed the answer materially.
- Tasks that succeeded but took too many turns or tool calls.
Redact without destroying the signal
Production traces often contain private data. Redaction should remove sensitive values while preserving the relationships that matter for evaluation. If the task depends on a join key mismatch, the redacted trace still needs mismatched keys. If the failure depends on a policy exception in free text, the redacted text still needs the exception shape.
A good redaction process keeps event order, tool type, error type, evidence markers, approval requirements, and final answer structure. Replace secrets and user identifiers, but do not flatten the trace into a generic prompt.
- Keep timestamps or relative ordering.
- Keep tool names and outcome categories.
- Mask identifiers consistently so relationships remain testable.
- Store redaction notes next to each case for reviewer auditability.
Convert traces into task contracts
A trace becomes an eval when it has an explicit contract. The contract should define the user goal, required evidence, allowed and forbidden tools, success criteria, budgets, and known failure modes. This lets the same case run repeatedly across models, prompts, or agent frameworks.
Do not encode only the final answer. Encode the behaviors that must happen along the way. If the original incident happened because the agent skipped a policy document, the eval should require evidence from that document. If it failed because approval was bypassed, the eval should include an approval precondition.
- Required observations before planning or writing.
- Expected artifacts or answer fields.
- Forbidden shortcuts, tools, or claims.
- Turn, retry, search, token, and cost budgets.
- Verification events before completion.
Use expert review where it matters
Not every check should be automated. Deterministic checks are excellent for presence, order, budget, forbidden actions, and exact artifacts. Expert review is better for whether a recovery strategy was appropriate, whether an answer was sufficiently nuanced, and whether the agent respected business context.
The right pattern is layered. Run deterministic checks first so reviewers spend time on judgment rather than bookkeeping. Then capture expert decisions in structured labels so the suite improves over time.
- Use deterministic scoring for trace mechanics.
- Use expert labels for domain correctness and judgment.
- Record reviewer rationale, not only pass or fail.
- Convert repeated reviewer notes into new automated checks.
Keep the suite alive
A production-derived eval suite is not a one-time project. It should change as the product changes. New tools add new failure modes. New models change behavior. New customers bring new edge cases. Treat the suite like a reliability asset with ownership, review cadence, and release impact.
The most useful suites become part of the operating rhythm. They run before model upgrades, prompt changes, retriever changes, and agent framework changes. They also run after incidents to confirm the failure has become a durable regression test.
- Review new failures weekly or during release readiness.
- Track pass rate and pathology rate by task family.
- Retire cases only when the product behavior no longer exists.
- Keep private suites separate from public demos and marketing examples.