XFS Metadata corruption detected at xfs_attr3_leaf_write_verify

[Stockley, Jonathan]

Hi,
I just ran into this error while testing an OpenStack SWIFT deployment.

[130004.933449] XFS (loop1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x468d0c8
[130004.936209] XFS (loop1): Unmount and run xfs_repair
[130004.937477] XFS (loop1): First 64 bytes of corrupted metadata buffer:
[130004.939113] ffff880111ddd000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................
[130004.941242] ffff880111ddd010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........
[130004.943327] ffff880111ddd020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[130004.945393] ffff880111ddd030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[130004.947565] XFS (loop1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-vt3Z1H/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc0752c92
[130004.951692] XFS (loop1): Corruption of in-memory data detected.  Shutting down filesystem

Environment information:
Ubuntu Server 14.04 LTS
$ uname -a
Linux 3e2116e0-b4e8-4666-be70-5ddf9c9d9d2b 3.19.0-49-generic #55~14.04.1hf1533043v20160201b1-Ubuntu SMP Mon Feb 1 20:41:00 UT x86_64 x86_64 x86_64 GNU/Linux

I am able to reproduce the problem as follows:

  *   created a VM based SWIFT cluster
One HAProxy load balancing across two SWIFT Proxy vms accessing five SWIFT storage nodes, although it could probably be simplified to one proxy and 1 storage node.
  *   Using ssbench<https://github.com/swiftstack/ssbench> with the following scenario file:
{
  "name": "file upload only”,
  "sizes": [{
    "name": "files”,
    "size_min": 100000,
    "size_max": 100000
  }],
  "initial_files": {
    "files": 1
  },
  "container_count":10,
  "operation_count": 10000,
  "crud_profile": [50, 50, 0, 0],
  "user_count": 50
}
  *   Run ssbench-master with following command line:
./ssbench-env/bin/ssbench-master run-scenario -f scenario1.json -A "http://aa.bb.cc.dd:8080/auth/v1.0" -U “acct:user" -K key --workers 10 --delete-after 36000 -r 18000

Replace aa.bb.cc.dd with either IP of HAProxy or SWIFT Proxy. Replace acct:user with SWIFT account and username. Replace key with user’s key (password). The test will run for 5 hours and objects will expire after 10 hours, but the test deletes all objects at the end of the run.

In my two test runs the XFS failure occurred around 9 hours after the test was started.

It looks like I can reproduce the problem, albeit over an extended period of time.
What can I do to gather more info? Any debug options I can enable that might help?

Regards,
Jo Stockley.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20160722/95bbb3b9/attachment.html>