"Model-agnostic" answers the question of whether we're locked to one provider. It doesn't answer the more practical question clients actually ask us: given that you could build on almost anything, how do you decide what actually goes into my agent? The honest answer is that it follows a repeatable process, not a house favorite we reach for by default.
The three variables that actually matter
Every model evaluation we run comes down to three things, weighed against each other rather than in isolation. The first is reasoning quality on your actual product data — not a generic benchmark score, but how well a model handles your specific catalog, your policy exceptions, and the ambiguous phrasing real customers use. A model that scores well on public leaderboards can still misread a return policy with three seasonal carve-outs.
The second is response latency at your expected volume. A live chat widget on your storefront has a different tolerance for delay than an overnight email queue, and that tolerance gets a lot less forgiving during a traffic spike like a flash sale or a holiday launch. The third is cost per resolved conversation, not cost per token. A model that's cheap per call but needs three follow-up messages to resolve a ticket can end up more expensive than a pricier model that gets it right the first time.
How the evaluation actually runs
Before we recommend a model, we build a test set of representative conversations drawn from your discovery call and process map — the same exceptions, edge cases, and phrasing patterns your support team deals with every week. We run candidate models against that set, not a generic benchmark, and score them for accuracy, tone, latency, and cost.
The result isn't always the model you'd expect. Sometimes the strongest fit for a client is a smaller or cheaper open-weight model, not the most talked-about flagship release. A store handling mostly straightforward order-status and shipping questions rarely needs frontier-level reasoning to resolve them correctly — it needs speed and consistency at a cost that scales with volume. Reserving the heavier, more expensive model for the smaller share of genuinely complex conversations is often the better economic call, and our evaluation is what surfaces that instead of guessing.
When clients already have a preference
Not every decision starts from a blank slate. Some clients come to us with a model already chosen — because of a compliance requirement, a data-residency rule, an existing vendor relationship, or simply a brand preference for how a particular model's tone comes across. In those cases, the evaluation shifts from "which model" to "how do we get the best result from this one," and we engineer around that constraint rather than pushing back on it.
This is common among clients with strict data-residency requirements, who need an open-weight model like Llama, Mistral, or Qwen deployed inside their own infrastructure rather than called through a third-party API. It's a legitimate reason to choose a model, and it changes the engineering work — self-hosted deployment, different latency characteristics — but not the discipline behind the decision.
It's a decision we revisit
The model we recommend at launch isn't locked in forever. We track how your agent performs against real conversations after it goes live, and if a better-fit model becomes available, or your traffic and catalog change enough to shift the tradeoffs, a swap is scoped like any other re-engineering work. Your prompts, integrations, and evaluation history carry over, because they were never written in one vendor's dialect to begin with.
The goal isn't to chase whichever model is trending. It's to make a defensible, evidence-based call for your specific workload, and to be willing to revisit that call when the evidence changes.
What this looked like for two recent builds
A recent client running a home goods storefront needed an agent that could field a high volume of simple, repetitive questions — order status, shipping windows, return eligibility — during peak seasonal traffic. Our evaluation prioritized latency and cost per resolved conversation over raw reasoning depth, since the conversations themselves were rarely ambiguous. A fast, lower-cost model cleared our accuracy bar comfortably and kept per-conversation cost low enough to scale with their holiday traffic without a corresponding cost spike.
Around the same time, a client selling technical outdoor equipment needed an agent that could reason through spec comparisons, compatibility questions between products, and a more layered return policy tied to product condition. The evaluation there weighted reasoning quality far more heavily, even at a higher per-conversation cost, because getting a technical answer wrong was a worse outcome than paying slightly more to get it right. Two clients, two different answers, both arrived at through the same process rather than a default.