What is a cloud GPU?
A cloud GPU is a graphics processing unit rented from a cloud provider and reached over the network. The customer chooses a GPU type, how many to attach to a virtual machine, and for how long, and the provider bills by the hour, by the month, or by a multi-year reservation. Both the large general-purpose clouds and a growing set of specialist GPU providers offer them.
Cloud GPUs come in a few purchase forms. On-demand instances start within minutes and can be released at any time, at the highest hourly rate. Reserved instances commit the customer to one or more years for a lower rate and guaranteed availability. Spot or preemptible instances use spare capacity at a discount but can be reclaimed by the provider with little notice, which suits interruptible training jobs and rules out production inference. A fourth form is the dedicated bare-metal server, a whole physical machine leased by the month.
Why cloud GPUs matter
Cloud GPUs remove the biggest barrier to starting with AI, which is the hardware. A team can evaluate three models on real company data in a week and decide whether the project is worth pursuing before any capital is spent. They also fit uneven workloads, such as a quarterly retraining or a one-time pass over ten years of documents. The trade-off appears at steady state. A GPU that serves inference around the clock for several years is cheaper to own than to rent, and data sent to a cloud GPU has left the building, with all the contractual and compliance questions that follow. Many organizations start on cloud GPUs and move steady inference to owned hardware once volume and value are proven.
Cloud GPU in practice
A hospital analytics group wants to fine-tune a language model to summarize discharge notes in its own documentation style. The job needs four GPUs for about two weeks. The group rents them in a cloud region covered by its business associate agreement, trains on de-identified notes, and releases the instances. The finished model, now the hospital’s own, is loaded on a single GPU server in the hospital data center for daily use, where the full notes never leave the network.
What to watch for
- Availability varies by region and GPU type, and the most sought-after cards are often waitlisted or unavailable on demand.
- Idle instances bill at the full rate, so automate shutdown and set spending alerts from the first day.
- Confirm which regions and contract terms are covered by compliance agreements before any regulated data touches an instance.
- Compare rental against ownership using the hours the GPU will actually be busy, because that number decides which is cheaper.