
Google and Meta open-sourced TorchTPU at PyTorch Conference last week, and if you’re running PyTorch workloads on Google Cloud TPUs, there’s a deadline you need to know: PyTorch/XLA — the previous approach — is deprecated and gets archived on November 15. TorchTPU is its replacement. The migration takes one line of code. Here’s what changed and what you need to do.
What TorchTPU Is
TorchTPU is a native PyTorch backend for Google Cloud TPUs, built jointly by Google and Meta. It plugs into the PyTorch dispatcher at the PrivateUse1 level — the same mechanism that makes GPUs and CPUs first-class devices. There are no wrapper tensors, no JAX runtime dependency, and no mark_step() calls. TPUs are now a torch.device like any other.
The old PyTorch/XLA workflow looked like this:
import torch_xla.core.xla_model as xm
device = xm.xla_device()
model = model.to(device)
xm.mark_step() # forget this and your workload silently stalls
The new TorchTPU workflow:
model = model.to("tpu")
If you use Hugging Face Transformers v5, it’s even more familiar:
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", device_map="tpu")
TorchTPU is v0.2.0 on PyPI, requires Python 3.11+, and ships under Apache 2.0 across four repositories: TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU, and TorchTitan.
Why Meta’s Involvement Matters
Meta created PyTorch in 2016. At the PyTorch Conference keynote, Meta engineer Zongwei Zhou announced that Meta plans to move larger-scale LLM workloads onto TPUs using TorchTPU. That’s not a partnership press release — it’s a hardware diversification signal from the organization that owns PyTorch’s trajectory. When Meta bets on TPU workloads, developers notice.
Three Execution Modes, One That Actually Matters
TorchTPU ships with three execution modes. Two are what you’d expect from any eager-mode framework: Debug Eager (synchronous, bounds checking, for debugging only) and Strict Eager (the default asynchronous mode, closest to GPU semantics). The third mode is where the performance story lives.
DeferAndFuse is a hybrid eager/compiled mode. Instead of dispatching every operation immediately, it defers and fuses multiple ops into larger chunks before a single TPU dispatch. The result: 50% or more faster than Strict Eager in Google’s tested workloads, with lower peak HBM memory usage as a bonus. It builds its fusion graph until it hits a materialization point — a .to("cpu") call, a .item(), anything that forces a result back to host — then dispatches the whole fused graph at once.
This matters because it addresses the main complaint about XLA’s lazy execution model without requiring developers to reason about graph boundaries manually.
Inference Infrastructure: vLLM and SGLang Go Native
Both vLLM and SGLang have TorchTPU-native versions in the four-repo release. The inference stack includes TPU Sync, an open-source C++ library that transfers KV cache data on background threads by reading directly from PyTorch device buffers — no staging copy through host CPU memory. This enables disaggregated prefill and decode across multiple hosts and host KV offloading without the transfer overhead that typically punishes large-context inference workloads.
At scale, TorchTPU has been validated on 8,192 Ironwood (TPU v7x) chips with near-linear scaling. Training Qwen 3 8B at that scale reached 52% model FLOPs utilization. In a more directly comparable test, TorchTitan on a 64-chip TPU v7 pod hit 1,828 tokens per second in training — 94% of JAX/MaxText performance. That parity number is more meaningful than any marketing claim: it tells you the gap between the PyTorch-native path and Google’s own optimized JAX stack is now single-digit percentage points.
The Caveats Worth Knowing
The “one line change” narrative is accurate for eager-mode workloads. It overstates the situation for torch.compile users. When you compile, TorchTPU routes through Dynamo to StableHLO and then to XLA’s compilation backend — the XLA compiler stack does not go away, only the Python frontend does. Developers who hit XLA compilation errors will still need to understand something about how XLA works.
Custom CUDA kernels have no TPU equivalent path yet. If your workload depends on hand-written CUDA, you’re staying on GPU for now. And TPU availability is still gated by Google Cloud quotas — this is not commodity hardware you can spin up on demand the way H100s are available across dozens of providers.
The November 15 Deadline
PyTorch/XLA is deprecated now, archived November 15. That gives existing XLA users roughly three weeks to start planning a migration. TorchTPU’s documentation includes a migration guide, and the official PyTorch blog covers the technical architecture in depth. Google’s announcement covers hardware compatibility — v5e, v5p, v6e, and v7/Ironwood are all supported. Hugging Face published a separate integration post showing Transformers v5 support across the 5,000 most-downloaded models, with 96.5% of tests passing.
For most PyTorch developers, TorchTPU is a genuine improvement over XLA: easier to debug, familiar device semantics, and an inference stack that matches production requirements. Whether it’s worth migrating your GPU workloads to TPU is a separate cost-benefit question. But if you’re already on PyTorch/XLA, you don’t have a choice — the archive date is fixed.

