I use debian and have apt configured to remount /usr read-write prior
to installing packages and then remount it read-only again after its
done, however today the /usr partition ended up deadlocked and the box
had to be reset.
i was still able to gather some information about what was going on,
what ps revealed is that a process started in the background by one of
the debian package postinst scripts (update-menus) had issued a
rm <list of files in /usr> command at the very same time as the
mount -o remount,ro /usr was run, both were stuck in a D state.
files already opened/cached from /usr could still be accessed, but
attempting to access anything uncatched (say ls /usr) would result in
that process going into a D state, as you would expect the box
deteriorated into a more and more crippled state fairly quickly.
also after having to reset the box a couple files were completly
replaced with nulls, this is not really unexpected in most of the
cases i found (files open read/write probably being updated when i had
to finally reset (i tried a clean shutdown first). however
/etc/inetd.conf was also completly replaced with nulls, which i find
quite odd since 1) i don't have inetd even running, and 2) it has not
been touched or should have even been accessed in the machines entire
uptime.
also one shared library from the xlibs package (libXm or so) was
missing, however this was not one of the packages being upgraded.
one the next boot i don't think /usr required any recovery, xfs_repair
and xfs_check report no problems. my own verification tests show no
other corruption there. also there were no log entries or kernel
messages, and no filesystem shutdown.
/usr /var / and /home are all separate partitions.
kernel is 2.4.18 with the split patches, on the powerpc arch.
i doubt i could reproduce this again if i tried. from my observation
of the rm and mount processes simultaniously deadlocked it smells to
me like a very tight race condition in the kernel somewhere.
--
Ethan Benson
http://www.alaska.net/~erbenson/
pgpakUfTEOoJD.pgp
Description: PGP signature
|