"It works on my machine" has never been an acceptable standard for shipping software that talks directly to customers, and it's an especially bad one for an AI agent, where a confident-sounding wrong answer can do more damage than an obvious bug ever could. Every agent we build goes through the same staged pipeline before it's allowed anywhere near a live storefront.
What the evaluation set actually contains
Before a build is tested, we assemble a test set from real conversation history and the process map built during onboarding — not synthetic, tidy examples written to make the agent look good. It includes the awkward phrasing, the multi-part questions, and the edge cases specifically flagged during process mapping as places where things tend to go wrong.
We run the agent against that set repeatedly during development, scoring it on accuracy, tone, and whether it escalates appropriately when a question falls outside its scope. A build doesn't move to the next stage until it clears a threshold on all three, not just raw correctness — a technically accurate answer delivered in the wrong tone can be just as damaging to a brand as a wrong one.
Staging before storefront
Once a build clears evaluation, it goes into a staging environment connected to a sandbox version of your store's data — real product structure and order patterns, but not live customer traffic. Our team runs adversarial testing here: deliberately trying to confuse the agent, push it past its scope, or get it to take an action it shouldn't. This is also where we test the handoff logic specifically — making sure escalation to a human happens cleanly, with the right context attached, rather than dropping a customer's conversation history at the handoff point.
Only after staging does an agent go live, and even then it launches with tighter monitoring for the first two weeks than it will have permanently — more frequent review of transcripts, faster response if something looks off, and a lower threshold for us to step in and adjust.
Why this discipline matters more here
A bug in most software fails loudly — a broken button, an error page, something a QA process catches easily. An AI agent's failures are quieter and more dangerous: a plausible-sounding wrong answer that a customer believes and acts on. That's why our testing standard weighs "how convincingly wrong can this be" as heavily as "how often is this right," and why staging always happens against sandbox data instead of testing new behavior directly against real customers.
What happens after launch
Testing doesn't stop once an agent goes live. Every transcript where a customer had to rephrase, escalate unexpectedly, or expressed frustration gets reviewed, and genuine gaps become new entries in that agent's evaluation set — the same set used before any future change is pushed. That means the agent's test coverage only grows over time, built from real conversations instead of our best guess at what customers might ask.
This also governs how we handle model updates and prompt changes after launch. Nothing gets pushed to a live agent without first passing the same evaluation set it launched with, plus whatever new cases have been added since. It's slower than pushing a quick fix directly to production, but it's the only way we're comfortable making changes to something already talking to your customers.
The threshold we won't launch below
We don't publish a single universal accuracy number, because the right threshold depends on what's at stake in a given conversation — a wrong answer about a return window is recoverable in a way a wrong answer involving a payment dispute isn't. Higher-stakes conversation categories carry a stricter bar and, often, a mandatory human handoff regardless of how confident the agent is. Getting that calibration right, category by category, is as much a part of the testing phase as accuracy scoring itself.