By ByteBot
ZStack open-sourced GPUWorks today under Apache 2.0. The platform manages GPU clusters for AI inference without Kubernetes, ships as a 40 MB download, and in a physical hardware test drove 1,003 GPU servers while delivering 69,000 tokens per second — the control plane used 9 CPU cores and 1.6 GB of RAM. The code is live at github.com/flitai-io/gpuworks.
The Kubernetes Problem Nobody Admits
Every major GPU inference platform — KServe, Ray Serve, Triton with autoscaling — assumes you already have a working Kubernetes cluster. That is a significant prerequisite. It means a cluster operator, YAML debt, and a Knative or KubeRay dependency before the first model even loads. For teams that want to self-host AI inference without spinning up a full container orchestration stack, the options have been thin.
GPUWorks takes a different approach. Install from a single command, open a browser, onboard GPU nodes, pick a model, and deploy. No controllers to configure, no admission webhooks to debug. ZStack claims a 60-second path from installation to a running inference endpoint.
The Numbers That Make This Interesting
The benchmark ZStack published is worth reading carefully. One GPUWorks Server managed 1,003 physical GPU servers. It spun up more than 500 inference replicas in four minutes and sustained 69,000 tokens per second across the cluster. The control plane consumed 9 CPU cores and 1.6 GB of RAM.
That overhead-to-throughput ratio is unusually lean. The platform is built around a small Server binary that handles scheduling, model lifecycle, and replica routing — Workers run on the GPU nodes and do the actual inference. There is no distributed consensus layer, no etcd cluster, no pod scheduler running alongside your models.
Model Sleep and Wake: Where the Cost Savings Live
GPUWorks manages model sleep and wake as a first-class operation. When a model sits idle, it releases GPU resources. When a request arrives, it wakes the model. That sounds straightforward until you factor in multi-GPU models — those need all participating GPUs to coordinate simultaneously. GPUWorks treats a multi-GPU model group as a single atomic unit, so wake and sleep happen without orphaned GPU allocations.
Wake latency from L1 memory is around four seconds. DeepSeek-V4.1-Flash wakes in 3.6 seconds. Compare that to a cold start — loading weights from NFS on eight H20 GPUs takes 16 to 17 minutes. The wake speed is roughly 200 times faster than cold. For teams running many large models with uneven traffic patterns, the GPU utilization improvement compounds quickly.
Three scheduling policies are available: priority (high-priority models stay resident), yield-on-idle (release resources when no requests arrive), and scheduled-rotation (time-based allocation for predictable workloads).
Hardware Support Beyond NVIDIA and AMD
GPUWorks supports NVIDIA and AMD out of the box. It also supports Ascend (Huawei), T-Head PPU (Alibaba), and Hygon DCU — Chinese AI accelerators that most Western open-source inference platforms do not natively handle. All hardware types appear in the same management plane, whether the nodes are on bare metal, virtual machines, or cloud containers across on-premises, private cloud, or public cloud environments.
For global development teams running heterogeneous infrastructure or operating in environments where NVIDIA supply is constrained, this is a practical advantage worth noting.
Four Hundred Model Profiles Ready to Deploy
The platform ships with approximately 400 model profiles and over 1,000 pre-configured launch configurations. Each profile bundles the container image, inference engine, weights, and runtime parameters into a single reusable unit. Selecting a model from the catalog and deploying it is a browser operation — not a YAML authoring exercise.
The catalog covers standard frontier models and a range of open-weight alternatives. If the model you need is not in the catalog, you can add a custom profile through the same interface.
Getting Started
The repository is at github.com/flitai-io/gpuworks under Apache 2.0. Installation is a single command from the README. GPUWorks Server runs on Linux and needs network access to the GPU nodes. The official announcement covers the deployment architecture in more detail, and Tech Insider has a solid technical breakdown of the benchmark methodology.
ZStack has 5,000+ enterprise customers across 36 countries and GPUWorks has been part of its commercial product stack. Opening the code under Apache 2.0 brings the community into the platform’s development at a moment when inference infrastructure is the most contested layer in AI deployment. Worth watching — and worth running in a lab environment this week.
