-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
ISSUE: teaparty file system read only for about nine hours (0012Z-0931Z 19-04-2025).
SUBSYSTEMS AFFECTED: anything that requires changing the contents of
the file system. This includes, but is not limited to, receiving email
and logging the sending of email. It did not seem to affect teaparty's
ability to serve static content, so DNS should have been unaffected.
DETAILS: teaparty has been essentially read-only for about nine hours.
At 0012Z today the root file system ran into the inability to store a
directory leaf checksum, so to preserve the integrity of the stored data
it decided to remount its own file system read-only, which prevented
any further changes beong made to the file system.
It turns out this is harder to detect than one might have thought
(though you can be sure I'll be adding a monitoring check for it shortly)
so I didn't spot it for about half an hour. We're currently in the US,
and it was very late at night on the 18th when the issue was noticed, so
to avoid doing anything precipitate that might mean the whole thing was
down until we get back in two weeks, we decided to sleep on the problem.
This morning (local time) I ran a command that compacts all the existing
directory structures, which is the recommended fix, held my breath. and
rebooted.
It came back fully read-write at 0931Z. Any email that people were
trying to send should have queued up in the interim, and will now be
delivered as other mail servers notice that teaparty is back. As far
as I can tell no email was lost, and nearly no data; the exception is
one user who I will contact directly, who lost about 200 mail files when
the file system was fixed.
I'm really sorry about this, and will be reading more about the problem
today as I have time, to find out if there's any routine maintenance
that can be done to prevent a recurrence.
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE35XTJ2Rqq4XkGD1cpc4IT7x26mMFAmgDdPEACgkQpc4IT7x2
6mMwMA//f7JPrhc7eT8SjpCxB6Qw0Lg+h3UDI7U/5c7JStmDQ5PC6xdgJxKylCT5
N4m/yu/O4FhV/w5xV+WbfVGhuPufLrbqX6gbINXXTHpsMsowqCWExUExkQ8lt5q7
nGtnXBY+STjhGFD27Cmhw1ADZh1lv5tJrEFCX3pEAjvzUBAV/+2iYAKudq9l1tYO
bzAsJDpyWTY/N4Gcd0KmDUP9PeaPdaq/2vpbPEJbRibQcLu4CgoXT97xZ5WSHD4+
aa2Xp0neGrL/ViKLVlH5hoBN7QlBazRkzi6GgNBH+WM+sUp+NVO+nSFl/uqxPnHI
8acgemL2lqWnuhv9amos2AM8LT0t08vEZvQvmkCaBIYQzJsQNeIAKv5hRRfckPYz
1Q8CqijVCQNG0Qdu1oIsHIj0QyWDXpHWHMKF8wVMWdu24c0gmRtkk4ECRGKCOZHi
nIbS1ckPAoRSCA4J1SgTCkqX4xgJc4DSG0GxxwgoE/FiMVtLH+ceF3ThtHQxbpkX
+TieuMPbnB7qlQrQ9gc1Lwwv10HamXktJCYx+i6tctligYhL5+AJX89Uy0N2wWzC
TqgP8tJT/TcOp0MHHLO913dY5eKXQqEl7eb2XotObSsovfjKi7fhurOuwTAbRHl8
1Hgz9IJcc7AENNKvcSeDv53PSo/WtFUXahVq/+KroRDWpdC24Vc=
=c2+N
-----END PGP SIGNATURE-----
