[Top] [All Lists]

Re: ENOSPC and filesystem shutdowns

To: Bernard Chan <bernard@xxxxxxxxxxxxx>
Subject: Re: ENOSPC and filesystem shutdowns
From: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Date: Mon, 5 Sep 2011 03:47:34 -0400
Cc: xfs@xxxxxxxxxxx
In-reply-to: <CAKxAxjFgBd0Vt02m7FJKq_ZWNBBJ5B=N4u7kS69FKtOKT4UowA@xxxxxxxxxxxxxx>
References: <CAKxAxjFgBd0Vt02m7FJKq_ZWNBBJ5B=N4u7kS69FKtOKT4UowA@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Sun, Sep 04, 2011 at 02:09:49PM +0800, Bernard Chan wrote:
> We have an XFS filesystem (on LVM, probably doesn't matter anyway) that is
> 4TB running on CentOS kernel,

Isn't Centos based on RHEL and thus running either 2.6.9, 2.6.18 or
2.6.32-ish kernels?

> We searched and found this list, and a few patches around kernel
> 2.6.26-2.6.27 that seem to match our scenario. We were able to log the
> specific mkdir command that failed and confirmed it consistently fails that
> way that gives "no space left on device", while we did not reproduce the
> same issue mkdir in other directories with large inode numbers. We haven't
> tried patching or upgrading the kernel yet, but we will do that later.
> As the root cause of that patch points to a bug triggered by ENOSPC, we
> checked the inode numbers created for some directories and files with "ls
> -li" and some of them are pretty close to 2^32.
> So, we would like to ascertain if that is the cause for ENOSPC in our case,
> and does that mean 32-bit inodes are no longer adequate for us and we should
> switch to 64-bit inodes? Will switching it avoid this kind of shutdowns with
> successful writes in the future?
> And is it true that we don't need a 64-bit OS for 64-bit inodes? How can we
> tell if our system supports 64-bit inodes?

It doesn't.  On Linux XFS only supports inode64 on 32-bit systems since
Linux 3.0.

> Finally, although we all know that "df -i" is sort of nonsense on XFS, how
> can we get the output of 5% inode while having inode numbers that are close
> to 2^32? So what does that 5% exactly mean, or were I looking at inodes the
> wrong way?

It's based on the available space given that XFS can theoretically use
any inode block for data.

> Thanks in advance for any insights anyone may shed on this one.

I'd move off a 4.5-year old unsupposed kernel.  The real RHEL/Centos
kernel have fairly good xfs support these days if you want a backporting
option.  Even RHEL5 might have inode64 on 32-bit systems as it has a lot
of XFS updates backport, but in doubt I would recommend to move to
a RHEL6/Centos6 kernel at least.

<Prev in Thread] Current Thread [Next in Thread>