<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hi,</div>
<div>I just ran into this error while testing an OpenStack SWIFT deployment.</div>
<div><br>
</div>
<div>
<div>[130004.933449] XFS (loop1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x468d0c8</div>
<div>[130004.936209] XFS (loop1): Unmount and run xfs_repair</div>
<div>[130004.937477] XFS (loop1): First 64 bytes of corrupted metadata buffer:</div>
<div>[130004.939113] ffff880111ddd000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................</div>
<div>[130004.941242] ffff880111ddd010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........</div>
<div>[130004.943327] ffff880111ddd020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</div>
<div>[130004.945393] ffff880111ddd030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................</div>
<div>[130004.947565] XFS (loop1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-vt3Z1H/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffc0752c92</div>
<div>[130004.951692] XFS (loop1): Corruption of in-memory data detected. Shutting down filesystem</div>
</div>
<div><br>
</div>
<div>Environment information:</div>
<div>
<div>Ubuntu Server 14.04 LTS</div>
<div>$ uname -a</div>
<div>Linux 3e2116e0-b4e8-4666-be70-5ddf9c9d9d2b 3.19.0-49-generic #55~14.04.1hf1533043v20160201b1-Ubuntu SMP Mon Feb 1 20:41:00 UT x86_64 x86_64 x86_64 GNU/Linux</div>
</div>
<div><br>
</div>
<div>I am able to reproduce the problem as follows:</div>
<ul>
<li>created a VM based SWIFT cluster<br>
One HAProxy load balancing across two SWIFT Proxy vms accessing five SWIFT storage nodes, although it could probably be simplified to one proxy and 1 storage node.</li><li>Using <a href="https://github.com/swiftstack/ssbench">ssbench</a> with the following scenario file:<br>
{<br>
"name": "file upload only”,<br>
"sizes": [{<br>
"name": "files”,<br>
"size_min": 100000,<br>
"size_max": 100000<br>
}],<br>
"initial_files": {<br>
"files": 1<br>
},<br>
"container_count":10,<br>
"operation_count": 10000,<br>
"crud_profile": [50, 50, 0, 0],<br>
"user_count": 50<br>
}</li><li>Run ssbench-master with following command line:<br>
./ssbench-env/bin/ssbench-master run-scenario -f scenario1.json -A "http://aa.bb.cc.dd:8080/auth/v1.0" -U “acct:user" -K key --workers 10 --delete-after 36000 -r 18000</li></ul>
<div><br>
</div>
<div>Replace aa.bb.cc.dd with either IP of HAProxy or SWIFT Proxy. Replace acct:user with SWIFT account and username. Replace key with user’s key (password). The test will run for 5 hours and objects will expire after 10 hours, but the test deletes all objects
at the end of the run.</div>
<div><br>
</div>
<div>In my two test runs the XFS failure occurred around 9 hours after the test was started.</div>
<div><br>
</div>
<div>It looks like I can reproduce the problem, albeit over an extended period of time. </div>
<div>What can I do to gather more info? Any debug options I can enable that might help?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Jo Stockley.</div>
<div><br>
</div>
</body>
</html>