Bug Reports
Complete

Fixed ThinkWatch and ThinkFeeds filling their database disks because replication was holding on to write-ahead logs indefinitely

What happened

The high-availability setup on these two services keeps a replication slot per database replica so a replica that falls behind can catch up. A slot that stops being consumed makes the primary database retain every write-ahead log since that point — forever. Disk use grew steadily until the volumes were full.

Impact

ThinkWatch and ThinkFeeds both risked their databases going read-only or refusing writes entirely. Detected 2026-07-28 during a routine check, already in progress — the exact start was never established.

Resolution

Retention was capped on both services the same morning, then the same check and the same cap were rolled out across every database on the platform over the following day so no other service could reach the same state unnoticed. A follow-on gap in ThinkAuth's own high-availability setup was found during that sweep and closed as well.

0 Comments

Sign in to comment

No comments yet. Be the first to share your thoughts!