Provisioning Options
Google Cloud offers several platforms for provisioning GPU-accelerated clusters, each built for a different workload, level of control, and scaling need. This page covers the five most widely used platforms and how to match them to an AI/ML workload.
The control vs. management spectrum
The five platforms sit on a spectrum from fully controlled (you manage the machines) to fully managed (the platform hides the infrastructure). Moving right trades control for less operational overhead.
- Cluster Director - the most control; dense, topology-aware clusters of interconnected accelerators.
- Compute Engine (GCE) - individual GPU VMs or small clusters you manage directly.
- GKE - container orchestration for GPU workloads (Autopilot or Standard).
- Cloud Run - serverless, scale-to-zero GPU inference.
- Agent Platform - the most managed; a fully-managed ML platform.
Cluster Director: large-scale AI infrastructure
Cluster Director (formerly Hypercompute Cluster) deploys and manages vast numbers - up to tens of thousands - of interconnected accelerator and networking resources as a single, homogeneous unit. It is built for densely allocated, performance-optimized infrastructure for large-scale AI, ML, and high-performance computing (HPC), with built-in integrations for GKE and Slurm schedulers.
Compute Engine: flexible VM management
Compute Engine creates and manages individual VMs or smaller clusters of VMs with attached GPUs. Common uses:
- Running graphics-intensive workloads.
- Executing simulation workloads.
- Performing small-scale machine learning model training.
Deployment options for creating GPU-attached VMs:
| Option | What it does |
|---|---|
| Single VM | One VM for serving or single-node workloads. |
| Managed instance group (MIG) | A group of GPU VMs, leveraging the Dynamic Workload Scheduler (DWS). |
| Bulk | Create groups of GPU VMs in bulk. |
| GPU workstation | Set up virtual GPU-accelerated workstations. |
GKE: container orchestration for AI workloads
Google Kubernetes Engine (GKE) is widely used for AI/ML workloads, with robust GPU support as a core feature. It offers two modes for using GPU resources:
| Mode | Who manages the nodes | How GPUs are attached |
|---|---|---|
| GKE Autopilot | GKE fully manages the underlying infrastructure and node provisioning. | Declare the GPU resources directly in your workload's configuration. |
| GKE Standard | You keep more control over the cluster nodes. | Manually attach GPUs to nodes, or use custom compute classes to control node properties; workloads then use the allocated GPUs. |
GKE further improves GPU utilization with multi-instance GPUs (MIG) and time-sharing, maximizing efficiency and cost-effectiveness.
Cloud Run: serverless AI inference
Cloud Run can be configured with GPUs, making it an excellent choice for AI inference workloads such as large language models (LLMs), as well as compute-intensive non-AI use cases such as video transcoding and 3D rendering. It is particularly good for deploying and scaling LLM-backed applications.
Agent Platform: the managed ML platform
Agent Platform is a comprehensive, fully-managed platform for building, training, and deploying ML models and AI applications. GPU-enabled VMs boost performance in several ways:
| Capability | What the GPU accelerates |
|---|---|
| Accelerated custom training | GPU-enabled VMs in custom training environments, using GKE worker pools for scalable computation. |
| Integrated foundational models | Open LLMs and other foundation models used directly from the Agent Platform Model Garden. |
| Reduced prediction latency | Faster predictions from trained models, for faster real-time inference. |
| Enhanced development environments | Faster code execution in Agent Platform Workbench notebooks and Colab Enterprise runtimes. |
Serving an open-source model with GKE
Open models are publicly available, pre-trained foundational LLMs, offered through platforms such as Agent Platform, Kaggle, and Hugging Face. Some demand significant infrastructure to deploy - which is where GKE simplifies serving these large models.
The demo provisions a GKE cluster with NVIDIA L4 GPUs, serves the Gemma 2B LLM with the Hugging Face Text Generation Inference (TGI) toolkit, and deploys a Gradio web chat app to talk to the model:
- Create the cluster -
gcloud container clusters create-auto ml-cluster --release-channel rapid --region <region>provisions an Autopilot cluster. - Hugging Face token - create a Read token, store it as a Kubernetes secret (
kubectl create secret generic hf-secret --from-literal=hf_api_token=$HF_TOKEN). - Serve the model - apply the TGI deployment (single L4 GPU); the L4 fits models under 24 GB, and TGI can shard larger models across multiple GPUs.
- Expose it - a service fronts the TGI deployment with a TCP load balancer; Managed Prometheus scrapes TGI metrics every 30s.
- Chat UI - deploy the Gradio app, watch
kubectl get deployments gradiountil Ready, then open the service's external IP.
A single NVIDIA L4 is a great fit for serving models under 24 GB. Larger models need TGI sharding across multiple GPUs (for example two L4s for a Falcon 40B-class model).
Recap
Choosing a platform comes down to how much control you need versus how much infrastructure you want the platform to manage.
Each platform offers distinct benefits - from Agent Platform's fully-managed ML environment and Cloud Run's serverless inference to the deep control of Compute Engine and Cluster Director and the scalable orchestration of GKE. Understanding these options lets you pick the optimal solution for a workload, whether you prioritize ease of use, granular control, or massive scalability.