Fixed several services silently running on empty, freshly-created databases after an automated dependency update went half-applied
What happened
An automated dependency update bumped the shared database component used by several Think services. The update landed only partially: the version was raised in one file but not in the matching lock file, and the tooling quietly fell back to the old version instead of failing. The two versions name their database clusters differently, so the platform read the fallback as "the old cluster is gone, create a new one" and built fresh, empty clusters alongside the originals.
Impact
Eleven days, 2026-07-17 to 2026-07-29. The affected services kept answering, which is why nothing alerted — but some were answering from a new, empty database rather than the real one. It was found by accident while investigating unrelated storage capacity, not by monitoring.
Resolution
Every affected service was put back on its original database cluster and the lock files were regenerated so the version bump applies completely or not at all. The failure mode that made this invisible — a silent fallback instead of a hard error — was closed off at the same time.
0 Comments
Sign in to comment
No comments yet. Be the first to share your thoughts!
