16-Year-Old SQLite Bug Behind Tailscale's Mystery Outages
Tailscale has finally got to the bottom of a series of database corruption incidents that began plaguing its service late last year. The culprit: a bug hiding inside SQLite since version 3.7.0, released in July 2010. It took six months of investigation, collaboration with the SQLite maintainers, and a custom-built diagnostic tool to find it.
For the uninitiated, Tailscale builds peer-to-peer mesh networking on top of WireGuard. Each private network, or 'tailnet,' is managed through a SQLite database running on Tailscale's control plane servers. The company adopted SQLite as its primary database in 2022, citing the usual reasons: it's mature, well-understood, and has an excellent track record.
The trouble started in August 2024, when Tailscale's backup pipeline began flagging database corruption. The backups work by snapshotting the full SQLite file every few minutes and shipping it to S3. Corruption kept appearing, repeatedly, with no obvious pattern and no recent changes to any of the low-level code involved.
The team ruled out the usual suspects one by one. POSIX locking problems from close() calls? No. Memory mismanagement? No. Thread-safety misconfiguration? Also no. Each incident generated more diagnostics and crossed off another theory.
Eventually suspicion landed on SQLite's Write-Ahead Log system and its checkpointing process. WAL mode is a performance feature that temporarily buffers database writes in a separate log file before folding them into the main database via a checkpoint operation. In typical deployments this happens automatically and invisibly. Tailscale, however, was running manual checkpoints to keep its backup process fast and consistent, which put it firmly off the standard operational path.
To get a clearer view of what was happening during checkpoints, the SQLite team built a virtual file system shim capable of logging VFS activity in detail, funded by Tailscale. They waited for the next corruption event, captured the logs, and had their answer.
The bug, now formally called the WAL-Reset bug, is a data race between a checkpoint operation and a concurrent write transaction. If a write lands at a precise moment during a checkpoint, the checkpointing process incorrectly believes certain pages have already been flushed from the WAL into the main database file when they haven't been. Those pages are lost permanently. Pages that reference them still get written. The result is a quietly corrupted database.
The SQLite team's analysis confirms the bug can only surface when WAL mode is active and multiple connections are open against the same database file simultaneously, with reads and writes colliding at exactly the wrong moment. In practice, this makes it vanishingly unlikely under normal use. Tailscale's manual checkpoint approach was unusual enough to push it into the danger zone.
SQLite maintainers believe the race condition has been present since version 3.7.0, meaning it sat undetected for roughly 16 years. It's now patched, and the SQLite team is recommending users update, while noting the risk to anyone running a standard configuration is extremely low.
The broader lesson here is fairly pointed. SQLite is one of the most tested and widely deployed pieces of software on the planet. It is also, like any complex system, capable of surprising you the moment you start operating it outside the parameters most people use. Taking manual control of checkpointing was a reasonable engineering decision for Tailscale's use case. It just happened to be the exact combination of conditions needed to wake up a bug that had been dormant since 2010.
As Tailscale engineer Alex Chan put it: 'By taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.' That's a polite way of saying the bug was always there, waiting for someone unusual enough to find it.