What is self-hosted AI?
Self-hosted AI is AI software and models that an organization installs, runs, and maintains on servers it administers, instead of calling a vendor’s hosted service. The servers may sit in the company’s own building, in a colocation facility, or in a cloud account the company controls. What makes it self-hosted is who holds administrative access and operates the software, rather than where the hardware is.
In practice this means downloading open-weight model files, running them with an inference server, and operating the surrounding pieces: the vector database for retrieval, the application that users see, monitoring, backups, and upgrades. The organization is responsible for all of it, in the same way it is responsible for a self-hosted database or email server.
Self-hosted AI compared with hosted models
A hosted model is run by a vendor and reached over the internet. The vendor handles scaling, patching, and model upgrades, and the customer pays per token or per seat. Self-hosting trades that convenience for control. The organization chooses the model, pins its version, decides what is logged, and keeps every prompt inside its own network. It also inherits the operational work: GPU drivers, security patches, capacity planning, and the occasional model that fails to load after an upgrade. Self-hosted AI is the usual way to achieve private AI, and on-premises AI is the subset of self-hosting where the servers are physically in the organization’s facilities.
Self-hosted AI in practice
A mid-sized law firm needs to search and summarize matter files that are protected by privilege. Sending client documents to an outside service would require client consent and a careful review of the vendor’s terms. Instead the firm’s IT team installs an open-weight model and a retrieval stack on a GPU server in its data center. Attorneys query matter files through an internal web application, documents never leave the firm, and the IT team schedules model upgrades during quiet weeks after testing on a sample of past matters.
What to watch for
- Budget for operations staff time, because the model is the easy part and the patching, monitoring, and upgrades continue for years.
- Check the model license, since some open-weight models restrict commercial use or use by companies above a certain size.
- Decide who approves model upgrades and how they are tested, so a quietly changed model does not alter the behavior of a validated workflow.
- Treat the inference endpoint as an internal production service with authentication, rate limiting, and backups of its configuration.