What is model routing?
Model routing is a layer in an AI application that examines each request and decides which model should handle it. Instead of one model serving every purpose, the application has several: a small, fast model on the company’s own hardware, a larger open-weight model on a GPU server, and perhaps a hosted model behind an API. The router applies rules, a classifier, or both to choose among them and returns the answer in a common format. The logic usually lives in an AI gateway that also handles authentication, logging, and rate limiting.
Routing decisions turn on four dimensions.
- Task type. Classifying a support ticket needs a different model than drafting a contract clause.
- Cost. A small model handles simple traffic so the larger, more expensive model is reserved for hard cases.
- Latency. Interactive requests go to the fastest model that meets the quality bar; overnight batch jobs can use a slower one.
- Data sensitivity. Requests containing personal, health, or financial data stay on a model inside the company’s environment; requests about public information may go to a hosted service.
Why model routing matters
No single model is best on every axis. The largest models are slow and costly per token, and sending every request to one is wasteful when most are simple. Routing lets a company use a hosted frontier model for the few cases that need it while running the rest on hardware it controls, which keeps costs predictable and regulated data inside. It also reduces vendor dependence, because a model can be swapped at the router without changing the applications that call it.
Model routing in practice
A healthcare billing company runs an assistant for its coding and appeals staff. Requests that include patient records, which is most of them, go to an open-weight model on the company’s own GPU servers. Requests to rewrite a generic appeal template or explain a public payer policy may go to a hosted model, and a small local model answers simple lookups in under a second. Every routing decision is logged with the data classification that drove it, for compliance review. In Kirk Tech Solutions engagements, the data sensitivity rule is set first and treated as a hard constraint, with cost and latency optimized inside it.
What to watch for
- A router that judges sensitivity with a model can misclassify, so back it with deterministic rules based on the source system and data classification.
- Different models answer the same prompt differently, so test prompts against every model in the pool and version them per model.
- Keep routing logic out of individual applications, or every change to the model pool touches every application.