By ByteBot
A new open-source project hit Hacker News front page this morning claiming something that sounds made-up: run an 80-billion-parameter language model in 4.3 GB of RAM on a Mac, or a 35B model on an iPhone 17. It is not marketing copy. The project is called Swiftlet, and the trick is real — it is just clever engineering applied to a property of the model most people gloss over.
The Trick: You Are Not Actually Running 80 Billion Parameters
Qwen3-Next-80B-A3B is a sparse Mixture-of-Experts model. The “A3B” in the name means only about 3 billion parameters are active on any given token. The router picks 10 of 512 expert networks per layer; the other 502 sit idle. Compute per token is roughly equivalent to a 3B dense model — about 26 times cheaper than a true dense 80B.
The catch with traditional deployments is the “memory paradox”: even though only 3B params activate per token, you still have to load all 80B into VRAM so the router has all options available. That means 40 GB or more of GPU memory just to start the model.
Swiftlet breaks this constraint. It keeps only the dense backbone — attention weights, projections, routers, embeddings — resident in RAM. That is roughly 2.5 GB. The expert weights live on SSD in a custom .qpack container format built for single-call reads. When the router decides which experts to use, Swiftlet fetches exactly those via a single pread call per expert. An LFU-plus-recency cache keeps frequently-accessed experts warm, hitting 43–70% cache rates in practice. Peak RAM stays at 4.3 GB.
What It Actually Runs
Two models are supported today:
| Model | Disk | Peak RAM | Speed (M5 Mac) | iPhone 17 |
|---|---|---|---|---|
| Qwen3.6-35B-A3B (4-bit) | 18 GB | 2.6 GB | 7–11 tok/s | ~1 tok/s |
| Qwen3-Next-80B-A3B (4-bit) | 42 GB | 4.3 GB | 4.5–5 tok/s | No |
The 35B is already shipping in Priv AI, an iOS app from the same developer. This is not a GitHub demo — it is in the App Store. The 80B stays on Mac for now, requiring Apple Silicon and macOS 14 or later.
The Speed Caveat (And Why It Does Not Kill the Story)
One token per second on iPhone 17 is slow. Real-time chat at that pace is uncomfortable. However, the 35B at 7–11 tok/s on an M5 Mac is fully usable for coding assistance, document summarization, and writing. Four-to-five tok/s for the 80B is workable for non-interactive tasks. You are not replacing a cloud API for latency-sensitive workflows.
There is also a more fundamental limitation the README is upfront about: “these models chat and write like large models but recall facts like small ones.” Only 3B parameters active per token means factual recall behaves more like a 3B model than an 80B one. Dense models or cloud APIs will outperform it on obscure factual questions.
Moreover, what Swiftlet offers is a different set of trade-offs. Your data never leaves the device. No API keys. No per-token billing. No rate limits. For regulated data — health records, legal documents, proprietary source code — those trade-offs often outweigh the speed difference entirely. ByteIota covered the broader economics of this in Open-Weight AI 2026: When Self-Hosting Beats the API; the on-device case is a logical next step down that curve.
Why Developers Should Pay Attention
Swiftlet ships with an OpenAI-compatible server mode running on loopback. Any client that speaks OpenAI’s chat completions API works without modification — point it at localhost instead of api.openai.com and your tooling carries over.
There is also an architectural detail worth noting for long-context work: 75% of the Qwen3-Next layers use Gated DeltaNet linear attention with a fixed-size recurrent state. A standard transformer KV cache grows with every token in context, which eventually causes out-of-memory failures on constrained devices. Gated DeltaNet’s fixed-state design keeps memory flat regardless of conversation length — an important property when processing long documents on a device with 8 GB total unified memory.
Getting started on macOS requires three commands:
git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release
.build/release/swiftlet-repack --from-hf Leonickson/Qwen3.6-35B-A3B-qpack --output ~/models/qwen3.6-35b.qpackRequirements: Apple Silicon (M1 or later), macOS 14+, and 18–42 GB of SSD space depending on the model. The Apache 2.0 license means you can embed it in commercial products.
The Bigger Signal
Swiftlet is not the only project doing SSD expert streaming. Flash-MoE, SwiftLM, and ssd-llm have all explored similar territory. What is different here is native Swift and Metal — not a llama.cpp wrapper — iOS native support, an App Store app already shipping, and validation against reference implementations rather than “it seems to generate something reasonable.”
The trajectory is clear. MoE architectures keep getting more parameter-efficient. Apple Silicon’s memory bandwidth keeps improving. SSD speeds on modern iPhones are fast enough to make streaming a practical memory tier. What felt like a research curiosity two years ago is becoming a real deployment option. The frontier model on your laptop is no longer hypothetical — it is 4.3 GB and a pread call away.
