Tailscale suffered 19 SQLite database-corruption incidents over six months. The failures did not line up with a recent deployment, a particular shard, a customer feature, load, or time of day, and the team could not reproduce them synthetically. The most useful clues were stranger: data committed by one transaction sometimes vanished before a later transaction, while checkpoint metrics claimed that more pages had been copied than the WAL contained.

The architecture was deliberately simple. A single Go process exclusively accessed the SQLite database for each control-plane shard, matching SQLite's single-writer strengths. Tailscale did depart from the common path in one important way. To produce fast, consistent backups, it manually controlled checkpoints and ran them aggressively instead of leaving the schedule entirely to SQLite. The configuration was documented and supported, but it exercised a rare timing window much more often than an ordinary deployment.

A new debugging tool built by SQLite's developers finally exposed the failure. The tmstmpvfs shim wraps the virtual-filesystem layer and records detailed page writes and checkpoint changes. After another production corruption, its trace revealed a data race between a write transaction and a WAL reset. At a precise moment, the checkpoint process could believe pages had reached the main database when they had not. Those pages were then permanently lost while references such as index entries survived, leaving the file inconsistent.

SQLite added a checkpoint check that detects whether another thread reset the WAL. The first release containing the fix, SQLite 3.52.0, created a separate false-corruption problem for stale expression indexes after a subtle text-to-floating-point conversion change. That release was withdrawn. Version 3.51.3 carried the WAL-reset fix without the conversion change, while Tailscale reduced timestamp precision to integer seconds to avoid ambiguity in its expression index.

The team wanted positive evidence, not merely a quiet period. It patched its driver to warn whenever a write and WAL reset overlapped. Two months later, the exact collision occurred and triggered the warning, but the database remained intact. Another four months passed without an incident by the time of publication. That observed near miss offered much stronger confirmation than the absence of corruption alone.

Community discussion also highlighted the productive relationship behind the fix: Tailscale purchased professional SQLite support and funded the open-source VFS tracing shim that isolated the race. The full engineering account explains the recovery work and investigation, while the Hacker News thread examines connection and checkpoint details. The lasting lesson is not to distrust SQLite, but to instrument and rehearse recovery whenever reliable technology is operated outside its most heavily tested path.

#database

A 16-Year-Old SQLite WAL-Reset Race Caused Tailscale's 19 Corruptions