|
Hi,
I just ran into this error while testing an OpenStack SWIFT deployment.
[130004.933449] XFS (loop1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x468d0c8
[130004.936209] XFS (loop1): Unmount and run xfs_repair
[130004.937477] XFS (loop1): First 64 bytes of corrupted metadata buffer:
[130004.939113] ffff880111ddd000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
[130004.941242] ffff880111ddd010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
[130004.943327] ffff880111ddd020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[130004.945393] ffff880111ddd030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[130004.947565] XFS (loop1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-vt3Z1H/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffc0752c92
[130004.951692] XFS (loop1): Corruption of in-memory data detected. Shutting down filesystem
Environment information:
Ubuntu Server 14.04 LTS
$ uname -a
Linux 3e2116e0-b4e8-4666-be70-5ddf9c9d9d2b 3.19.0-49-generic #55~14.04.1hf1533043v20160201b1-Ubuntu SMP Mon Feb 1 20:41:00 UT x86_64 x86_64 x86_64 GNU/Linux
I am able to reproduce the problem as follows:
- created a VM based SWIFT cluster
One HAProxy load balancing across two SWIFT Proxy vms accessing five SWIFT storage nodes, although it could probably be simplified to one proxy and 1 storage node. - Using ssbench with the following scenario file:
{
"name": "file upload only”,
"sizes": [{
"name": "files”,
"size_min": 100000,
"size_max": 100000
}],
"initial_files": {
"files": 1
},
"container_count":10,
"operation_count": 10000,
"crud_profile": [50, 50, 0, 0],
"user_count": 50
} - Run ssbench-master with following command line:
./ssbench-env/bin/ssbench-master run-scenario -f scenario1.json -A "http://aa.bb.cc.dd:8080/auth/v1.0" -U “acct:user" -K key --workers 10 --delete-after 36000 -r 18000
Replace aa.bb.cc.dd with either IP of HAProxy or SWIFT Proxy. Replace acct:user with SWIFT account and username. Replace key with user’s key (password). The test will run for 5 hours and objects will expire after 10 hours, but the test deletes all objects
at the end of the run.
In my two test runs the XFS failure occurred around 9 hours after the test was started.
It looks like I can reproduce the problem, albeit over an extended period of time.
What can I do to gather more info? Any debug options I can enable that might help?
Regards,
Jo Stockley.
|