Same endpoint, wildly different jobs
"Classify this ticket as billing or technical" and "draft a nuanced response to an angry enterprise customer" arrive at the same chat completions endpoint, in the same wire format, from the same SDK. Nothing about the request itself tells a naive integration that one of these is a lookup and the other needs real judgment — so a system with exactly one model configured treats them identically, which means it's either overpaying on the easy one or underdelivering on the hard one, permanently, by construction.
Classify first, then score, then pick
NeuroRoute's router runs a task classifier over every request — ten-plus task types (general, conversation, summarization, factual Q&A, classification, extraction, code, translation, creative writing, and more), each with a confidence score. That classification feeds a scorer that evaluates every candidate model in the current catalog — more than 30 models across Anthropic, Google Vertex, OpenAI-compatible endpoints, xAI, Groq, DeepInfra, AWS Bedrock, OpenRouter, and self-hosted vLLM — on quality for that specific task, current cost, current latency, and context-window fit, then picks the model that clears your configured quality bar at the lowest cost.
Quality isn't a single static number frozen at launch, either. A durable per-model, per-task quality profile is seeded from a compiled baseline and then re-aggregated hourly from real thumbs-up/down feedback over a rolling window, persisted so it survives a restart. A model that's actually underperforming on your traffic — not on a generic benchmark, on your traffic — gets down-weighted without anyone editing a config file.
Six strategies, because "cheapest" and "best" aren't opposites
- Cheapest — always the lowest-cost model that still clears the quality bar.
- Best-quality — quality-first, cost second, for work where a wrong answer is expensive to unwind.
- Task-aware — the classifier's confidence directly modulates how much weight cost gets, so an easy, high-confidence request is priced like the lookup it is.
- Fastest — optimizes for latency when a user is waiting on the response in real time.
- Balanced — the default, a middle path between all three axes.
- Cascade — cheapest-eligible-first with escalation only on a hard failure, so the fallback path costs nothing extra on the vast majority of requests that never need it.
Fusion strategy adds a seventh option for the genuinely hard tail: fan the prompt to the top-N scored candidates in parallel, and a cheap judge model picks the single best answer — verbatim, never merged or rewritten — so quality goes up on hard prompts without ever risking a synthesis hallucination, and without paying frontier rates on the 90% of traffic that was never going to need it.
The weighting isn't fixed — it moves with confidence
Task-aware routing's headline idea is simple, but the detail is what makes it safe. The default scoring profile is quality-dominant — quality weighted at 0.75, cost at only 0.10 — because that's the right default for a code question or a legal summary. But for a small set of genuinely easy task classes (general chat, plain conversation, summarization, factual Q&A), the weighting shifts to quality 0.50 and cost 0.30, and only when the classifier's own confidence in that task label is at least 0.60. Below that confidence threshold, the request keeps the quality-dominant weighting regardless of which task it was guessed to be — because a low-confidence "factual_qa" guess might really be a disguised code question, and routing that cheaply on a bad guess would silently reintroduce the exact overpay-or-underdeliver problem this feature exists to solve. Translation tasks are deliberately excluded from the easy set for the same reason: translation reads as mechanically simple, but a cheap model can degrade badly the moment the language pair isn't a high-resource one, in a way that's invisible to whoever's watching cost, not fluency.
The catalog is wide on purpose
"Pick the cheapest model" only saves real money if there's real price spread to pick from. NeuroRoute's live catalog spans Anthropic's Claude line, Google Vertex's Gemini tiers (Pro down to Flash and Flash-Lite), OpenAI-compatible endpoints, xAI, Groq, AWS Bedrock, and — doing a lot of the heavy lifting on the cheap end — DeepInfra's open-weight catalog: models like GLM-5.2, Kimi K2.7, Llama 3.3 70B, and DeepSeek-V3, each individually quality-scored per task rather than dumped into the router as one undifferentiated "cheap" bucket. A DeepInfra key with a working OpenRouter fallback also gets automatic cross-provider failover — if DeepInfra itself has a provider-level outage, in-flight requests remap to equivalent models on OpenRouter rather than hard-failing. And for workloads that need to stay on infrastructure you control, self-hosted vLLM models slot into the exact same catalog, scored and routed to on the same terms as every hosted provider.
The catalog also doesn't go stale quietly. A weekly discovery sync checks every provider for new and changed models, and a separate probing pass actually attempts a live call against anything sitting in a pending state to confirm a provider will really serve it before it's ever promoted into production routing. On the other side of that lifecycle, a model that stops responding for 48 continuous hours is automatically flagged for decommission rather than left silently absorbing traffic it can no longer serve — reviewed by an admin, never auto-removed, and never hard-deleted even once decommissioned, since usage and billing history still needs to reference the model that generated it.
Walking one request through the whole pipeline
Send "What's the capital of France?" through the API with strategy set to task-aware. The classifier tags it factual_qa at high confidence — this is an unambiguous, textbook example of the task type, so the confidence score clears the 0.60 gate easily. That flips the scoring weights to the easy-task profile: quality 0.50, cost 0.30. The scorer then walks the catalog, filters out anything that fails a hard policy check (a PII-restricted allowlist, a denied model on that key), and scores what's left on quality-for-factual-qa, current cost, current latency, and whether the model's context window comfortably fits the request. A small, cheap model with a solidly-adequate factual-qa quality score wins comfortably over a flagship model whose marginal quality edge is invisible on a question with exactly one correct, short answer. The response comes back in milliseconds, with a cost around a fraction of a cent — and if you pull that request's ID through the routing-explain endpoint afterward, you can see every candidate that was considered, its score on each axis, and why the winner won, rather than just being told to trust the number on the invoice.
You can see the reasoning, not just trust it
Every routing decision is explainable after the fact: a routing-explain endpoint returns which models were even considered, their scores on each axis, the confidence of the classification, and the actual provider cost of the response — not a black box that says "trust us, this was cheaper." If a customer wants to independently verify that a strategy change actually improved outcomes, an evaluation framework lets them benchmark a routing-config version, a hard model pin, or a strategy override against their own prompt suites — quality score, cost, and latency per test case — before promoting anything to production traffic.
The point isn't that every request should go to the cheapest model. It's that the decision of which model handles which request should be made per-request, on real signal, instead of once, by whoever configured the integration first and never revisited it.