What is an inference endpoint?
An inference endpoint is the URL through which applications reach a deployed model. A request carrying the prompt, the model name, and settings such as maximum length goes to the endpoint over HTTPS, and the response carries the generated text or prediction. Behind the address sits an inference server, either run by a vendor as a hosted model or run by the company itself. Endpoints are usually protected by an API key or single sign-on and often sit behind an AI gateway that adds logging, routing, and rate limits.
Why inference endpoints matter
The endpoint is the seam between a model and the business systems that use it. A distributor’s ERP add-on that drafts supplier emails does not know whether the model behind the endpoint is hosted or self-hosted. When the company moves from a hosted API to an open-weight model in its own data center, the change can be a new base URL and model name in configuration. Endpoints are also where control is applied: who may call them, how often, which models are allowed, and what is logged. A private endpoint reachable only from inside the company network is a common requirement in regulated industries.
What to watch for
- Keep endpoint URLs and keys in configuration and a secrets manager, never in application code.
- Set client timeouts and retry rules deliberately, because generation can take many seconds.
- Put every endpoint, including vendor ones, in the AI inventory so usage is visible.