[Top] [All Lists]

Re: problem after growing

To: Eric Sandeen <sandeen@xxxxxxxxxxx>
Subject: Re: problem after growing
From: Rémi Cailletaud <remi.cailletaud@xxxxxxxxxxxxxxx>
Date: Thu, 14 Feb 2013 17:37:41 +0100
Cc: xfs-oss <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <511CFCBD.504@xxxxxxxxxxx>
References: <511BC78B.6070205@xxxxxxxxxxxxxxx> <511BCB41.4060804@xxxxxxxxxxx> <511BCD11.20907@xxxxxxxxxxxxxxx> <511BCFB6.8000309@xxxxxxxxxxx> <511BD0FB.2070401@xxxxxxxxxxxxxxx> <511BD2D1.9010906@xxxxxxxxxxx> <511BD6ED.6030006@xxxxxxxxxxxxxxx> <511BEE8B.8000400@xxxxxxxxxxx> <511BF3B6.9040502@xxxxxxxxxxx> <511C07C6.9080003@xxxxxxxxxxx> <511C9E9C.8080200@xxxxxxxxxxxxxxx> <511CB0C3.9090507@xxxxxxxxxxxxxxx> <511CFC06.2030103@xxxxxxxxxxxxxxx> <511CFCBD.504@xxxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130116 Icedove/10.0.12
Le 14/02/2013 16:03, Eric Sandeen a écrit :
On 2/14/13 9:00 AM, Rémi Cailletaud wrote:
Le 14/02/2013 10:39, Rémi Cailletaud a écrit :
Le 14/02/2013 09:21, Rémi Cailletaud a écrit :
Le 13/02/2013 22:38, Eric Sandeen a écrit :
On 2/13/13 2:12 PM, Eric Sandeen wrote:
On 2/13/13 1:50 PM, Eric Sandeen wrote:
On 2/13/13 12:09 PM, Rémi Cailletaud wrote:
Le 13/02/2013 18:52, Eric Sandeen a écrit :

Did the filesystem grow actually work?

# xfs_db -c "sb 0" -c "p" /dev/vg0/tomo-201111
magicnum = 0x58465342
blocksize = 4096
dblocks = 10468982745

That looks like it's (still?) a 38TiB/42TB filesystem, with:

sectsize = 512

512 sectors.

How big was it before you tried to grow it, and how much did you try to grow it 
by?  Maybe the size never changed.
Was 39, growing to 44. Testdisk says 48 TB / 44 TiB... There is some chance 
that it was never really growed.
At mount time it tries to set the sector size of the device; its' a hard-4k 
device, so setting it to 512 fails.

This may be as much of an LVM issue as anything; how do you get the LVM device 
back to something with 512-byte logical sectors?  I have no idea...

*if* the fs didn't actually grow, and if the new 4k-sector space is not used by 
the filesystem, and if you can somehow remove that new space from the device 
and set the LV back to 512 sectors, you might be in good shape.
I dont either know how to see nor set LV sector size.  It's 100% sure that 
anything was copied on 4k sector size, and pretty sure that the fs did not 
really grow.
I think the same blockdev command will tell you.

Proceed with extreme caution here, I wouldn't start just trying random things 
unless you have some other way to get your data back (backups?).  I'd check 
with LVM folks as well, and maybe see if dchinner or the sgi folks have other 
Sigh... No backup (44To is too large for us...) ! I'm running a testdisk 
recover, but I'm not very confident about success...
Thanks to deeper investigate this...
First let's find out if the filesystem actually thinks it's living on the new 
What is the way to make it talk about that ?
well, you have 10468982745 4k blocks in your filesystem, so 42880953323520 
bytes of xfs filesystem.

Look at your lvm layout, does that extend into the new disk space or is it 
confined to the original disk space?
Seems it does not : lvm map shows 48378494844928 bytes, 1304432738304 on the 4K 

lvm folks I talk to say that if you remove the 4k device from the lvm volume it 
should switch back to 512 sectors.

so if you can can convince yourself that 42880953323520 bytes does not cross 
into the newly added disk space, just remove it again, and everything should be 

Unless your rash decision to start running "testdisk" made things worse ;)
I tested this.  I had a PV on a normal 512 device, then used scsi_debug to 
create a 4k device.

I created an LV on the 512 device&   mounted it, then added the 4k device as 
you did.  growfs failed immediately, and the device won't remount due to the sector 
size change.

I verified that removing the 4k device from the LV changes the LV back to a 512 
sector size.

However, I'm not 100% sure how to remove just the 4K PV; when I did it, I did 
something wrong and it reduced the size of my LV to the point where it 
corrupted the filesystem.  :)  Perhaps you are a better lvm admin than I am.
How did you remove the pv ? I would tend to use vgreduce, but I'm a bit (a lot, 
in fact) scary about fs corruption. That's why I was wondering about pvmove'ing 
extents on a 512K device
Or may a vgcfgrestore be safer ? Should I ask lvm folks ?
I tried a test as you suggest using scsi_debug.

2 PV, one 512 and one 4096 bytes sector.

after adding the 4K device, growfs fail, and partition wont remount. I tried a 
vgcfgrestore and vgreduce, but it does not mount : same error...
XFS (dm-5): device supports 4096 byte sectors (not 512)
In that case I think you must not have actually (completely?) removed the 4k 

That's it! After vgchange-ing un/available, lv mounts !

Following steps reproduce the "bug", considering we already have one pv on /dev/sdc (512 sectors device) :

- create a virtual 4k scsi device and create a pv on it :
# modprobe scsi_debug sector_size=4096 dev_size_mb=256
# pvcreate /dev/sdd

- create a vg with both pv, and create an lv on sdc (I specified exact extents count of sdc) :
# vgcreate vgtest  /dev/sdc /dev/sdd
# lvcreate -n lvtest -l 3759 vgtest

- mkfs, mount :
# mkfs.xfs /dev/vgtest/lvtest
# mount /dev/vgtest/lvtest /mnt/tmp

- the bad thing : lvextend and growfs (should not lvm or xfs check this sector size stuff ?):
# lvextend -l+40 /dev/vgtest/lvtest
# xfs_growfs /mnt/tmp
(fail with xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed)

- the scary part :
# umount /mnt/tmp
# mount /dev/vgtest/lvtest /mnt/tmp
mount: function not implemented
# tail -1 /var/log/messages
Feb 14 16:41:52 hamaika kernel: [ 481.055422] XFS (dm-5): device supports 4096 byte sectors (not 512)

- the huge relief (restoring *before* lvextend) :
# vgcfgrestore -f /etc/lvm/archive/vgtest_00029-84595486.vg vgtest
# vgchange -a n vgtest
# vgchange -a y vgtest
# mount /dev/vgtest/lvtest /mnt/tmp

yippee, my datas are back !!

Should I submit a bug report ? On LVM, XFS, both ?

However, a great thanks for your help... I learned some stuff about lvm and xfs today ;)


I think you'll need to seek help from LVM people in order to proceed...  I'm 
sure it's possible to safely and completely remove the newly added space, but I 
don't know how.


Rémi Cailletaud - IE CNRS
3SR - Laboratoire Sols, Solides, Structures - Risques
BP53, 38041 Grenoble CEDEX 0
Tél: +33 (0)4 76 82 52 78
Fax: +33 (0)4 76 82 70 43

<Prev in Thread] Current Thread [Next in Thread>