By ByteBot
Tailscale published a detailed post-mortem this week on how 19 production database corruptions over six months led engineers to a data race hiding in SQLite’s WAL checkpoint code since July 2010. The bug — now officially called the WAL-Reset bug — exists in every version of SQLite from 3.7.0 through 3.51.2, and it was fixed in SQLite 3.51.3 on March 13, 2026. If your application runs SQLite in WAL mode with concurrent connections and you haven’t verified your version, stop and check right now.
Six Months, 19 Corruptions, Zero Answers
Tailscale’s corruption events started in August 2025. Their transaction logging pipeline was the first to flag the problem clearly: data written and committed by one transaction was inexplicably invisible to later transactions. They ran PRAGMA integrity_check against S3 backups and confirmed the worst. Recovery time improved from over an hour to under an hour across those months — but the corruptions kept coming. Nineteen incidents in all.
What made this exceptionally difficult was Tailscale’s non-standard checkpoint strategy. They take manual control of SQLite’s checkpoint process and run it aggressively to keep S3 backups lean. Standard configurations let the library manage checkpointing automatically, which — it turns out — made the buggy code path far less likely to activate. Tailscale’s approach practically guaranteed they’d hit the race. To trace it, they funded a custom debugging tool called tmstmpvfs: a VFS shim that wrapped SQLite’s virtual filesystem layer to capture traces precise enough to catch the corruption mid-flight.
The SQLite WAL Bug That Hid for 16 Years
WAL mode works by appending committed transactions to a separate .wal file, then periodically checkpointing — copying those pages back into the main database file. The checkpoint thread reads the WAL header, noting the current walSalt (a counter that resets when the WAL file cycles). If a writer resets the WAL with a new salt value at precisely the wrong moment, the checkpoint doesn’t detect the change. It updates shared memory fields claiming pages were copied when they weren’t. The next checkpoint skips those pages, treating them as already done. They’re permanently gone — no error thrown.
Ubuntu/Canonical’s engineers modeled this exact behavior using TLA+ formal verification and discovered the corruption scenario in approximately 20 model-checking states. The fix required a single comparison in the checkpoint function: if walSalt changed since the checkpoint began, a concurrent reset occurred — abort and retry. That’s the entire patch. Sixteen years of latency for one comparison. The Ubuntu team also confirmed that dqlite (used in LXD) is not affected, because it acquires an exclusive write lock before any checkpoint, making the race structurally impossible.
Are You Affected? Check Now
The affected range is SQLite 3.7.0 (July 2010) through 3.51.2 (January 2026). The fix is in 3.51.3 and all later releases. However, many applications bundle their own SQLite rather than using the system package — so a system-level OS update is not sufficient. You need to verify the version your application actually loads.
-- In any SQLite-connected session:
SELECT sqlite_version();
-- Must return 3.51.3 or higher to be safe
-- Confirm WAL mode is active:
PRAGMA journal_mode;
-- Returns "wal" if you're in the affected mode
-- Check for existing corruption:
PRAGMA integrity_check;
-- "ok" means no corruption detectedOpenAI’s Codex team already pinned their bundled SQLite to the fixed version via PR #27992. SUSE issued security advisory SUSE-SU-2026:21095-1. Python users can run import sqlite3; print(sqlite3.sqlite_version) to check the exact loaded version. If you’re on 3.51.2 or earlier and running WAL mode with concurrent connections, upgrade. Corruption is unlikely but not theoretical — Tailscale experienced it 19 times across six months of production traffic.
Key Takeaways
- The SQLite WAL-Reset bug affects versions 3.7.0–3.51.2; the fix is in 3.51.3+. Upgrade immediately if you use WAL mode with concurrent connections.
- The bug is a data race between a checkpoint thread and a writer resetting the WAL salt — tight timing, permanent data loss, no error thrown.
- Standard, single-connection SQLite usage is safe in practice. Risk is highest with custom or aggressive checkpointing alongside multiple concurrent connections.
- Bundled SQLite in runtimes (Python, Node.js, Electron) may not update automatically — verify your application’s actual loaded version, not just your OS package.
- TLA+ formal verification found this bug in 20 model states; SQLite’s 92 million lines of tests did not. “Boring tech” used non-standardly carries non-boring risk.
