What is a canary deployment?
A canary deployment is a release method in which a new version runs beside the current one and receives only a small portion of live requests at first. If the new version misbehaves, few users are affected and traffic is routed back to the old version. If error rates, latency, and business metrics stay healthy, the share of traffic grows in steps until the new version carries everything.
For AI systems the canary is usually a new model version, a changed system prompt, or an updated retrieval index. Routing happens at the AI gateway or load balancer, and model monitoring reports each version separately. A canary protects a release, while A/B testing and champion-challenger testing compare versions deliberately over a longer period.
Canary deployments in practice
An online furniture retailer updates the language model behind its customer service assistant. The new version handles one conversation in twenty. The team compares resolution rate, escalations to human agents, and customer ratings between the two groups. The new model resolves slightly more conversations but makes more refund promises it is not authorized to make, so the rollout pauses while the prompt and tool permissions are corrected.
What to watch for
- Choose the metrics before the release, and include business outcomes as well as technical health.
- Run the canary long enough to see a representative mix of requests, including quiet and busy hours.
- Where the new version can change a system of record, keep the approval workflow in place for both versions.