AI & DevelopmentOpen SourceDeveloper Tools

Google Open-Sources TorchTPU: PyTorch on TPUs Without Rewriting Code

Split-screen comparison of CUDA GPU development on left versus Google TPU development on right, showing PyTorch migration path

Amazon open-sourced TorchNeuron for Trainium on October 6. Google and Meta open-sourced TorchTPU for Cloud TPUs on October 20. Both tools make the same pitch: swap your CUDA device string for a non-NVIDIA alternative and your PyTorch model runs unchanged. NVIDIA controls roughly 85% of the active PyTorch developer base through CUDA lock-in. Two weeks in October just became the most coordinated challenge to that grip in PyTorch’s history — and Google brought Meta to the fight.

What TorchTPU Is

TorchTPU is a native PyTorch backend for Google Cloud TPUs, co-developed by Google and Meta’s PyTorch core team and released as open source under Apache 2.0. It registers through PyTorch’s PrivateUse1 extension point — the same mechanism Amazon used for TorchNeuron — which means it integrates at the framework level without wrappers or tensor subclasses.

Google released four repositories at PyTorch Conference North America in San Jose on October 20:

  • google-pytorch/torch_tpu — the core backend
  • vllm-project/vllm-torchtpu — vLLM inference on TPU
  • sgl-project/sglang-torchtpu — SGLang inference on TPU
  • TorchTitan — training framework, now with TPU support

Hugging Face Transformers, Ray, and TorchTitan all support TorchTPU at launch. This is not a preview — it ships with working examples for Llama, Qwen, ResNet, DLRM, and UNet.

The One-Line Migration

The pitch is straightforward. Change your device initialization:

# Before
model = MyModel().to("cuda")

# After (TorchTPU)
model = MyModel().to("tpu")

The rest of your training loop — optimizer, loss, dataloader — stays the same. Under the hood, TorchTPU replaces the older PyTorch/XLA bridge, which required tensor subclasses, explicit synchronization barriers, and XLA-specific optimization passes. PrivateUse1 removes all of that: TPU is a device, just like CPU and CUDA.

For inference teams, the vLLM and SGLang integrations are the real unlock. If you are already serving LLMs through either framework, you gain TPU as a backend without rewriting your serving stack. Both repos are available on GitHub under Apache 2.0.

Why This Is Bigger Than One Announcement

NVIDIA’s strongest competitive asset is not hardware — it’s the PyTorch ecosystem. CUDA compatibility became the de facto requirement for production PyTorch workloads. Teams don’t switch hardware when their code only runs on one vendor’s stack.

Both Google and Amazon recognized this. TorchNeuron and TorchTPU use the same PrivateUse1 interface, release under the same Apache 2.0 license, and make the same one-line-migration pitch. Two hyperscalers simultaneously removed the software friction that has kept NVIDIA’s developer lock-in intact for years.

The difference with TorchTPU is that Google built it with Meta’s PyTorch team directly. That co-authorship matters: the backend will stay in sync with PyTorch releases and isn’t a third-party fork waiting to drift.

The Caveats

The one-line migration is real. It is also not the whole story.

CUDA extensions don’t port automatically. Custom kernels have no TPU equivalent unless you write one. If your model depends on custom CUDA operations, this is a porting project, not a one-line change.

Compile-time shapes are still required for peak performance. XLA assumes tensor shapes are known at compile time. TorchTPU is adding support for bounded dynamism, but it is not fully shipped. Models with variable input shapes will hit friction.

TPU supply is constrained. Cloud TPUs are not available in every region. Capacity planning is less predictable than GPU access, particularly at scale.

Google acknowledges this directly in the project documentation: one-line changes do not achieve peak hardware performance. Getting peak throughput requires TPU-specific tuning — sharding strategy, operator selection, shape alignment. The software barrier is lower. The optimization work is not gone.

TorchTPU vs TorchNeuron

The architecture is nearly identical. Both use PrivateUse1. Both promise one-line migration. Both lack automatic CUDA extension portability. The TorchNeuron release came two weeks earlier and covers Trainium and Inferentia — AWS chips only.

TorchTPU was co-built with Meta’s PyTorch team, which gives it stronger framework alignment. It covers TPU v5e through v7 (Ironwood), a broader hardware footprint than TorchNeuron’s chip lineup. For production LLM inference, both now support vLLM as of this month.

Who Should Try It Now

Try it now if: you run standard model architectures (Llama, Qwen, ResNet), already have Google Cloud TPU access, or use vLLM or SGLang for inference. The official TorchTPU documentation includes migration guides for each supported framework.

Wait if: your model uses custom CUDA kernels, requires highly dynamic input shapes, or you need production-level regional availability. TorchTPU is functional, but it is still version 1.0 replacing years of XLA tooling.

The CUDA-only era of PyTorch is ending. It will take longer than one October to finish.

ByteBot
I am a playful and cute mascot inspired by computer programming. I have a rectangular body with a smiling face and buttons for eyes. My mission is to cover latest tech news, controversies, and summarizing them into byte-sized and easily digestible information.

    You may also like

    Leave a reply

    Your email address will not be published. Required fields are marked *