By ByteBot
Perplexity has open-sourced Lily, the local inference engine powering its Mac Hybrid Compute feature — a Rust and Metal runtime that beats Apple’s MLX framework by 1.35x on decode throughput for Qwen3.6-35B-A3B, with no PyTorch or MLX anywhere in its execution path. The code is live in the pplx-garden repository today.
What Lily Actually Is
Lily is not a framework. It is not trying to be the next llama.cpp. It is a single-process runtime built for one model on one hardware family, and that constraint is the whole point.
The stack is three layers deep. A Rust runtime loads the model checkpoint and drives the generation loop — chosen because Rust gives memory safety without a garbage collector, which matters when you are managing large model weights across long generation sessions. Below that, hand-written Metal kernels execute the model directly on the GPU, bypassing any intermediate abstraction layer. On top, an OpenAI-compatible chat-completions API streams tokens to whatever client is calling it.
No MLX. No PyTorch. No framework overhead. That is the bet Perplexity made, and it paid off in tokens per second.
The Numbers
Benchmarked on a 40-core, 128 GB M5 Max MacBook Pro at batch 1, across ten prompt lengths from 256 to 128K tokens:
| Metric | Lily | MLX-LM | Speedup |
|---|---|---|---|
| Prefill (tokens/s) | 4,156 | 3,388 | 1.23x |
| Decode (tokens/s) | 170.0 | 126.4 | 1.35x |
Output quality was essentially unchanged: perplexity delta of 0.04%, and Lily selected the same top-ranked token as MLX-LM at 96.35% of positions. The speed gain is real; the quality loss is noise.
Why MLX Falls Behind
MLX is a general-purpose array framework. That generality is also its ceiling. Lazy evaluation and framework-level scheduling introduce overhead that does not exist in a dedicated inference runtime. When Lily handles prefill and decode as separate workloads — mapping Qwen’s specific operations directly to Apple Silicon’s compute and memory architecture — it is exploiting knowledge that MLX, by design, cannot assume.
This is not a knock on MLX. For running a variety of models experimentally, it is the right tool. But for production serving of a single model, you are paying a framework tax in every token.
Hybrid Compute and Privacy Gate
Lily exists because of Perplexity’s Hybrid Compute architecture, which went live for the Mac app on September 1. The idea: frontier models in the cloud handle planning, reasoning, and web search. The local model handles anything that should never leave your machine.
Alongside Lily, Perplexity also open-sourced Privacy Gate — a small on-device PII classifier that inspects outbound data before it reaches any cloud endpoint. It applies one of four outcomes: keep data local, mask sensitive spans, refuse the action, or ask for explicit user consent. The result is that a prompt like “summarize my bloodwork” can stay entirely on-device without the user needing to manually audit what gets sent where.
Open-sourcing the inference engine and the privacy classifier together is a deliberate signal: trust in local AI requires transparency about both what runs fast and what stays private.
How to Try Lily Now
The pplx-garden repository includes a standalone Lily demo with a minimal OpenAI-compatible HTTP API for greedy text generation. You do not need the Perplexity Mac app. You do need an Apple Silicon Mac with enough unified memory to load a 35B-parameter model — plan for at least 64 GB for comfortable operation.
The OpenAI-compatible endpoint means existing tooling that already calls a cloud API can point at Lily instead with no code changes beyond the base URL.
The Specialist Argument
Lily is interesting precisely because it refuses to generalize. The trend in open-source inference has been toward universal runtimes — llama.cpp runs hundreds of models, Ollama wraps that into a convenient interface, MLX handles the Apple Silicon side. Each layer of generality comes with a cost.
What Lily demonstrates is that when you know the exact model and the exact hardware, you can eliminate that cost entirely. The Perplexity engineering blog frames this as “on-device compute shouldn’t bottleneck Computer tasks” — meaning Lily’s job is to be fast enough that the local model never becomes the rate-limiting step in a hybrid workflow.
Expect more of this. As AI companies operationalize local inference at scale, the specialist runtime — tuned for one model, one architecture — will become a standard artifact alongside the model weights themselves. Lily is an early, well-executed example of that pattern. The benchmark data backs it up.
