On July 22, 2026, a Rust-based tokenizer called GigaToken hit the top of Hacker News with a claim that stopped LLM engineers mid-scroll: 989x faster than HuggingFace tokenizers. Marcel Rød’s one-person project processes text at 24.53 GB/s on a 144-core AMD EPYC — enough to tokenize the entirety of Common Crawl in under 7 hours. HuggingFace would need months. The 114-comment thread that followed wasn’t about whether the benchmarks were real. They are. The debate was about whether they matter.
Why GigaToken Is That Fast: SIMD, Caching, Rust
HuggingFace tokenizers run four steps on every input: normalize, pre-split, BPE merge, and post-process. Each step leans on regex engines. tiktoken is faster — roughly 2-3x — because it skips normalization and goes straight to a regex chunk before BPE. GigaToken replaces the regex entirely with SIMD instructions, processing multiple bytes in parallel at the CPU level. It also caches pretoken-to-token mappings. Token distributions in real text are highly long-tailed — a small set of pretokens covers most of what you’ll ever see — so caching turns repeated merge lookups from O(k log k) BPE operations into O(1) cache hits. Finally, GigaToken minimizes crossing the Python-Rust boundary by batching operations aggressively, eliminating the serialization overhead that other libraries pay thousands of times per document.
The result: benchmarks on Apple M4 Max (16 cores) show 8.79 GB/s for GPT-2, a 1,268x improvement over HuggingFace. The critical caveat is that these gains are specific to BPE-based models. SentencePiece families — Llama 1 and 2, T5, Gemma — use unigram tokenization rather than BPE, and GigaToken’s optimizations don’t transfer. Speedups on those families land in the low double digits rather than the hundreds.
import gigatoken as gt
# Drop-in replacement for HuggingFace tokenizers
tokenizer = gt.Tokenizer("Qwen/Qwen3-8B").as_hf()
tokens = tokenizer.encode_batch(["Hello world", "Another text"])When 989x Faster LLM Tokenization Actually Matters
The HN skeptics had a point. For standard GPU inference, tokenization is under 0.1% of wall time. GPU VRAM transfers, prefill, attention computation, and decode dwarf it. Switching tokenizers in a typical inference service will not move your p99 latency in any measurable way. If that describes your workload, GigaToken is not for you right now.
However, two categories of workloads change the calculation. First: pretraining and large-scale fine-tuning pipelines, where tokenization runs on CPU clusters before data ever reaches a GPU. At 24.53 GB/s, GigaToken turns batch preprocessing jobs from overnight processes into real-time operations. At this throughput on a 144-core EPYC, you can tokenize Common Crawl’s 130 trillion tokens in under 7 hours. That’s not a marginal improvement — that’s a different class of iteration speed.
Second: large-context inference for agentic workflows. As prompts grow — tool call histories, retrieved code, conversation chains — tokenization costs climb. Rød published measurements showing up to 40% TTFT reduction for GPT-OSS-120B and 18% for DeepSeek V3 on large-context requests. For inference routing systems processing millions of requests a day, that compounds. The HN discussion thread captures this debate well — it’s worth reading if you’re evaluating GigaToken for production.
Installation and Limitations
Installation is a single command:
pip install gigatokenThe HuggingFace compatibility mode uses .as_hf() to return an object with an identical encode_batch interface. Token output is byte-identical to HuggingFace Tokenizers in tested configurations — model weights and vocabulary are untouched. GigaToken 0.9.0 on GitHub supports 23 tokenizer families including Llama 3 through 4, Qwen 2 through 3, DeepSeek V3 and V4, Phi-4, GLM 5, and GPT-2.
The limitations are real. SentencePiece-based models see minimal gains. Windows support is incomplete (WSL recommended). Version 0.9.0 is beta — not yet 1.0. File sinks aren’t implemented. And the benchmark methodology isn’t perfectly apples-to-apples: GigaToken encodes entire unsplit files; HuggingFace is benchmarked on 100MB samples. The 989x figure is a ceiling, not a floor. Install it, test it against your specific workload before committing to production use. The PyPI package page has the full dependency and platform matrix.
Key Takeaways
- GigaToken is the fastest BPE tokenizer available, hitting 24.53 GB/s through SIMD pretokenization, pretoken caching, and minimal Python-Rust boundary crossings.
- For pretraining data pipelines, GigaToken is a genuine unlock — turning multi-day tokenization jobs into hours.
- For standard GPU inference, the speedup won’t move the needle. The math doesn’t favor switching unless you’re serving very large context windows at high volume.
- Install with
pip install gigatokenand migrate with the.as_hf()wrapper. SentencePiece-based models are excluded from significant gains. - Version 0.9.0 is beta. Test before committing, especially on non-x86 or Windows environments.
