All research
Evaluation·February 4, 2026

Evaluating AI Agents in Production: Beyond Offline Test Suites

How to combine offline evals, online signals, and Traces to know whether agents still work after you ship.

By Meebly Team·6 min read

Offline evals catch many regressions before promotion. Production is where traffic, data, and user language shift. Evaluating agents in production means measuring task success continuously—not waiting for a quarterly demo failure.

What "Good" Means for Agents

Define success metrics per App:

  • Task completion — did the user get the outcome they asked for?
  • Tool correctness — right Action, right parameters, recoverable errors
  • Safety — no policy violations; HITL respected for sensitive tools
  • Latency — time to first useful response and end-to-end run duration
  • Human burden — approval rate, escalation rate, re-ask rate

Accuracy of a single model reply is a weak proxy when the agent must call Actions.

Offline Suites Still Matter

Keep a golden set in a non-production Environment:

  • Happy paths with Screen Observer context
  • Ambiguous asks that should trigger clarification
  • Auth failures and tool timeouts
  • HITL-required Actions that must pause
test('refund path requires approval', async () => {
  const events = await runAgent({
    environmentId: 'sandbox',
    messages: [{ content: 'Refund order ORD-100 for full amount' }],
  });

  expect(events).toContainEqual(
    expect.objectContaining({ type: 'tool-call-approval', toolName: 'create_refund' })
  );
});

Run the suite on every instruction or Action schema change before Environment promotion.

Online Evaluation Patterns

Sampled human review

Review a random sample of Traces weekly. Score task success and note failure modes. Cheap, high signal, especially early.

Outcome labels from your product

If the agent opens tickets or completes checkouts, join run IDs to business outcomes. That is stronger than thumbs-up alone.

LLM-as-judge with guardrails

Automated judges help at volume but need rubrics and spot checks. Use them for tone and instruction adherence; prefer deterministic checks for "was HITL triggered" and "did the tool return ORDER_NOT_FOUND."

Shadow and canary agents

Run a candidate agent on mirrored traffic or a small cohort. Compare tool failure rates and completion metrics via Tool Monitoring before full promotion.

Instrument for Eval, Not Just Debugging

Traces should make scoring easy:

  • Final status and reason codes
  • Actions attempted and outcomes
  • Whether approval was required and granted
  • User clarification loops

If you cannot answer "did this run succeed?" from Trace fields, your online eval pipeline will stay manual forever.

Drift Detection

Watch for gradual degradation:

  • Rising tool failure rates after a backend change
  • Increasing HITL declines (agent proposing bad Actions)
  • Longer conversations for the same intents
  • New error codes appearing in Action responses

Alert on deltas by Environment and agent version, not only absolute thresholds.

Closing the Loop

Production findings should become offline cases:

  1. Find a failed Trace
  2. Sanitize and add it to the sandbox suite
  3. Fix instructions, Actions, or MCP schemas
  4. Re-run evals and promote

That flywheel is how agent quality compounds.

Conclusion

Evaluating agents in production combines offline regression suites, online outcome signals, and Traces that expose Action-level truth. Treat evaluation as part of Environment promotion and day-two operations—the same discipline you already apply to critical backend services.