XFS corruption - Ubuntu 14.04 VM with RDM

[Amir Soroka <amirsoroka@xxxxxxxxx>]

Hello,

We have a corruption issue. Would be happy to get a solution.

Configuration is Ubuntu 14.04 (generic) as a VM over ESXi 5.5, with 1.6 TB RDM - storage has a Cache battery, Raid 6, SSD disks.

During some heavy indexing activity we get a corruption as you can see below.

Some more information:

* Mount options: defaults,noatime,nodiratime,nobarrier,inode64,logbufs=8 0 0

* XFS was created like this: mkfs.xfs -f -d su=131072,sw=8 -i size=1024 /dev/sdb1

* Parted command was: /sbin/parted -s $i mklabel gpt mkpart /dev/sdb1 xfs 2048s 100%

[ Â 10.395575] XFS (sdb1): Mounting V4 Filesystem

[ Â 10.430107] XFS (sdb1): Ending clean mount

[ Â675.504000] XFS (sdb1): Metadata corruption detected at xfs_agf_read_verify+0x61/0x100 [xfs], block 0x1

[ Â675.504043] XFS (sdb1): Unmount and run xfs_repair

[ Â675.504062] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[ Â675.504218] XFS (sdb1): metadata I/O error: block 0x1 ("xfs_trans_read_buf_map") error 117 numblks 1

[ Â675.504257] XFS (sdb1): page discard on page ffffea003fbdc280, inode 0x86, offset 0.

[ Â677.186009] XFS (sdb1): Metadata corruption detected at xfs_agf_read_verify+0x61/0x100 [xfs], block 0x1

[ Â677.186052] XFS (sdb1): Unmount and run xfs_repair

[ Â677.186070] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[ Â677.186239] XFS (sdb1): metadata I/O error: block 0x1 ("xfs_trans_read_buf_map") error 117 numblks 1

[ Â677.186283] XFS (sdb1): page discard on page ffffea0000d8d480, inode 0x87, offset 0.

[ 4203.145905] XFS (sdb1): Metadata corruption detected at xfs_agf_read_verify+0x61/0x100 [xfs], block 0x1

[ 4203.145957] XFS (sdb1): Unmount and run xfs_repair

[ 4203.145976] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[ 4203.146132] XFS (sdb1): metadata I/O error: block 0x1 ("xfs_trans_read_buf_map") error 117 numblks 1

[ 4203.146171] XFS (sdb1): page discard on page ffffea002c1399c0, inode 0x60003c0c, offset 0.

[84997.832171] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.835828] XFS (sdb1): Unmount and run xfs_repair

[84997.836502] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.839848] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.841188] XFS (sdb1): Unmount and run xfs_repair

[84997.841872] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.845252] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.846523] XFS (sdb1): Unmount and run xfs_repair

[84997.847154] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.850467] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.851714] XFS (sdb1): Unmount and run xfs_repair

[84997.852319] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.855469] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.856718] XFS (sdb1): Unmount and run xfs_repair

[84997.857340] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.860366] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.861521] XFS (sdb1): Unmount and run xfs_repair

[84997.862117] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.865178] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.866356] XFS (sdb1): Unmount and run xfs_repair

[84997.866962] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.869987] XFS (sdb1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x110

[84997.871163] XFS (sdb1): Unmount and run xfs_repair

[84997.871760] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[84997.874798] XFS (sdb1): metadata I/O error: block 0x110 ("xfs_trans_read_buf_map") error 117 numblks 16

[84997.875947] XFS (sdb1): xfs_do_force_shutdown(0x1) called from line 315 of file /build/linux-lts-vivid-6tG2ln/linux-lts-vivid-3.19.0/fs/xfs/xfs_trans_buf.c. Return address = 0xffffffffc02596ab

[84997.888409] XFS (sdb1): I/O Error Detected. Shutting down filesystem

[84997.889017] XFS (sdb1): Please umount the filesystem and rectify the problem(s)

[84997.889643] XFS (sdb1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.

[85011.531433] XFS (sdb1): xfs_log_force: error -5 returned.

[85041.596418] XFS (sdb1): xfs_log_force: error -5 returned.

[85071.661447] XFS (sdb1): xfs_log_force: error -5 returned.

[85101.726398] XFS (sdb1): xfs_log_force: error -5 returned.

[85131.791376] XFS (sdb1): xfs_log_force: error -5 returned.

[85161.856372] XFS (sdb1): xfs_log_force: error -5 returned.

[85191.921373] XFS (sdb1): xfs_log_force: error -5 returned.

[85221.986341] XFS (sdb1): xfs_log_force: error -5 returned.

[85252.051322] XFS (sdb1): xfs_log_force: error -5 returned.

[85282.117357] XFS (sdb1): xfs_log_force: error -5 returned.

[85312.181319] XFS (sdb1): xfs_log_force: error -5 returned.

[85342.246276] XFS (sdb1): xfs_log_force: error -5 returned.

[85372.311273] XFS (sdb1): xfs_log_force: error -5 returned.

[85402.376250] XFS (sdb1): xfs_log_force: error -5 returned.

[85432.441269] XFS (sdb1): xfs_log_force: error -5 returned.

[85462.506208] XFS (sdb1): xfs_log_force: error -5 returned.

[85492.571203] XFS (sdb1): xfs_log_force: error -5 returned.

[85522.636192] XFS (sdb1): xfs_log_force: error -5 returned.

[85552.701213] XFS (sdb1): xfs_log_force: error -5 returned.

[85582.766164] XFS (sdb1): xfs_log_force: error -5 returned.

[85612.831140] XFS (sdb1): xfs_log_force: error -5 returned.

[85642.896133] XFS (sdb1): xfs_log_force: error -5 returned.

[85672.961128] XFS (sdb1): xfs_log_force: error -5 returned.

[85703.026104] XFS (sdb1): xfs_log_force: error -5 returned.

[85733.091083] XFS (sdb1): xfs_log_force: error -5 returned.

[85763.156070] XFS (sdb1): xfs_log_force: error -5 returned.

[85793.221079] XFS (sdb1): xfs_log_force: error -5 returned.

[85823.286041] XFS (sdb1): xfs_log_force: error -5 returned.

[85853.351019] XFS (sdb1): xfs_log_force: error -5 returned.

[85883.416016] XFS (sdb1): xfs_log_force: error -5 returned.

[85913.481014] XFS (sdb1): xfs_log_force: error -5 returned.

[85923.032422] XFS (sdb1): xfs_log_force: error -5 returned.

[85923.033429] XFS (sdb1): xfs_log_force: error -5 returned.

[85923.033601] XFS (sdb1): xfs_log_force: error -5 returned.

[86173.742817] XFS (sdb1): Mounting V4 Filesystem

[86173.780730] XFS (sdb1): Starting recovery (logdev: internal)

[86173.787173] XFS (sdb1): Metadata corruption detected at xfs_agf_read_verify+0x61/0x100 [xfs], block 0x1

[86173.788445] XFS (sdb1): Unmount and run xfs_repair

[86173.789072] XFS (sdb1): First 64 bytes of corrupted metadata buffer:

[86173.792242] XFS (sdb1): metadata I/O error: block 0x1 ("xfs_trans_read_buf_map") error 117 numblks 1

[86187.845410] XFS (sdb1): Mounting V4 Filesystem

[86187.881669] XFS (sdb1): Ending clean mount

Thanks,

Amir