GPU/TPU interoperability

Exam guide§2.1

Serving large language models is a constant trade-off between price, performance, and availability. Getting the best balance traditionally meant separate deployments per accelerator, more management, and higher cost. The goal here is a single deployment whose workload can intelligently switch between GPUs and TPUs - favouring TPUs for performance and falling back to GPUs for cost and availability.

Choosing the accelerator: a decision tree

Before you can make a workload portable, answer the first question: is your model code written for a GPU or a TPU? The tree branches from there, and both branches can cross over to the other accelerator or loop back.

GPU/TPU selection decision tree - each branch can cross to the other accelerator or dead-end at 'You're stuck'.

?DECISIONModel written for a TPU - which path?

Follow the left branch. TPUs are specialized for Google's frameworks and ideal for large-scale training.

1. Do you have access to the required TPUs? Yes → secure the TPUs and deploy. Done.
2. No access → can your current code run on a GPU? Many frameworks support both, so this is often possible. Yes → check GPU access.
3. No → can you convert the code to run on a GPU? This means refactoring the model. Yes → assess GPU access.
4. No, and no TPU access → you're stuck. Re-evaluate the model or your resource strategy.

Pick this when: TPU access → deploy | else can it run on / convert to GPU → assess GPU access | no path → stuck

?DECISIONModel written for a GPU - which path?

Follow the right branch - the more common case, with broader framework support.

1. Do you have access to the required GPUs? Yes → secure the GPUs and deploy. Most direct route.
2. No → can you use smaller GPUs? Less powerful GPUs sometimes suffice. Yes → secure those.
3. No → can your code run on a TPU? Yes → check TPU access (this loops into the TPU branch).
4. No → can you convert the code to a TPU? A significant effort. Yes → assess TPU access.
5. No, and no GPU access → you're stuck. Rethink the approach.

Pick this when: GPU access → deploy | smaller GPUs → deploy | runs on / convert to TPU → assess TPU access | no path → stuck

Demo: switching between GPU and TPU with vLLM

Rather than pick one accelerator up front, you can let GKE choose at scale time and keep one deployment portable across both.

≡FactsThe building blocks

• Custom compute classes define a priority order for node pools in a GKE cluster. Here: TPU nodes first (maximum performance), GPU nodes as a fallback (cost-effectiveness, extra capacity on demand).
• vLLM is a fast library for LLM inference and serving.
• A dual-container pod: because vLLM uses different base images for GPU vs TPU, the pod runs two containers. Whichever accelerator is present, that container's vLLM server starts; the other sleeps. Only the correct server is ever active for the underlying hardware.

Demo: GPU/TPU interoperability with vLLM on GKE

The demo deploys Mistral 8x7B (a popular MoE model) with vLLM onto two clusters: an A3 mega VM with H100 GPUs and a TPU VM with Trilliums. Comparing the two deployment YAMLs side by side, the only differences are:

≡FactsWhat changes between the GPU and TPU YAML

• The node selector.
• The image URL (GPU vs TPU base image).
• The pod resource configuration.

Everything else is identical - so an existing vLLM-on-GPU workload ports to TPU with just a few tweaks. Sending the same prompt to both endpoints returns the same response, because vLLM TPU automatically compiles the GPU checkpoints from Hugging Face: same weights, same results. The monitoring and Horizontal Pod Autoscaler manifests are also 100% identical, so Day-2 operations feel the same on either accelerator.

How the scaling behaves

Guided by the custom compute class, GKE scales up TPU nodes first to handle the initial load. As demand grows and TPU capacity is reached, GKE then scales up GPU nodes, keeping the service available and cost-optimized. For example, with the TPU pool constrained to one node, the first pod lands on the TPU node and subsequent pods spill over onto L4 GPU nodes.

⚠GotchaInteroperability costs some peak performance

There is an inherent trade-off between interoperability across hardware architectures and the peak performance of code optimized for one architecture. Interoperability via the XLA compiler often requires code changes. With vLLM on TPU (used in this demo) those changes are handled by the vLLM backend, but other serving libraries may require more effort.

≡FactsKey takeaways

• GKE + custom compute classes + vLLM enable intelligent, dynamic scaling of LLM workloads across both TPUs and GPUs in a single deployment.
• The setup prioritizes performance (TPU nodes first), then scales to cost-effective GPU nodes as demand rises or TPU capacity is reached.
• This keeps the service continuously available and cost-optimized while managing the interoperability-vs-peak-performance trade-off.