[Top] [All Lists]

Re: "No space left on device" from hell

To: Fran Tsao SantÃn <tsao@xxxxxxxxxxxxxxx>
Subject: Re: "No space left on device" from hell
From: Arkadiusz MiÅkiewicz <arekm@xxxxxxxx>
Date: Wed, 24 Sep 2014 10:54:05 +0200
Cc: xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maven.pl; s=maven; h=from:to:subject:date:user-agent:cc:references:in-reply-to :mime-version:content-type:content-transfer-encoding:message-id; bh=LxPvQ6LrmdUQwomvSDIvIWYKK8Rc+6OTcA0RxqI0Bqs=; b=mev7HS+fT1m9efnUqtdMc/2KNN8URVJv3HjkOYiU7Wl/zV4oqeH02kgx6bh9cAbr+T x4elMMVh0Lqu3ky7gFvBNpJUpV1IrOY/aN3Yxd9+1OaI6F0kjg3JoPUXPEfz/o5hADmf Mojj0b6YxSTo3MaUVB+UVbjStxvdQPE43w/jo=
In-reply-to: <CAM5tkX-mOO6EtJczQZC-23-vk7LwSczTL8CB6pgLTSLH0q94fg@xxxxxxxxxxxxxx>
References: <201409240903.46899.arekm@xxxxxxxx> <CAM5tkX-mOO6EtJczQZC-23-vk7LwSczTL8CB6pgLTSLH0q94fg@xxxxxxxxxxxxxx>
User-agent: KMail/1.13.7 (Linux/3.17.0-rc6; KDE/4.14.0; x86_64; ; )
On Wednesday 24 of September 2014, Fran Tsao SantÃn wrote:
> 2014-09-24 9:03 GMT+02:00 Arkadiusz MiÅkiewicz <arekm@xxxxxxxx>:
> > Hello.
> > 
> > 3.10.40 kernel and creating files causes "No space left on device". After
> > deleting some files I can create new files again until the problem
> > happens again.
> Are you sure the problem is the filesystem device? Maybe you are running a
> process that fills up another device i.e. shm.

Dave Chinner figured out the problem already (on irc).

Basically this fs was grown from 300GB to 600GB but due to bug (fixed by 
commit 9de67c3 ("xfs: allow inode allocations in post-growfs disk space")) 
that new space was never used for new inodes.

So ended up with full AGs 0-3 (xfs_db -r -c "freesp -s -a XX"  /dev/xyz shows 
that for each AG XX).

Fs was never unmounted after grows but fortunately mount -o remount,inode32 
... followed by mount -o remount,inode64 cured it without a need to reboot, so 
now new inodes are distributed to all AGs.

I still need to free some space in AGs 0-3 though (by moving data to new dirs 
allocated in new AGs; xfs_db -r -c "convert inode DIR_INODE_NR agno" /dev/xyz 
shows in which AG new dir got allocated).

There are more bugs hiding (like if ENOSPC on some AG happens it should try 
next AG instead of failing). Dave knows all that magic.

Big thanks to Dave!

Arkadiusz MiÅkiewicz, arekm / maven.pl

<Prev in Thread] Current Thread [Next in Thread>