<div dir="ltr"><div><div>The xfs shutdown error is always the same. (It happens about 20 times on about 50 servers during last half year)<br>Recently the shutdown happens on the cluster with 24 servers(dis 8 * replica 3) during rebalance. The average work load of this cluster is more than 3TB growth per day.<br>
The work load including normal fops, rsync jobs, video encoding/decoding, logging etc. through glusterfs native client of hundreds of machines. <br></div>The shutdown tend to happen when we run rebalance for the glusterfs cluster which I guess will trigger a lot of unlink operations?<br>
<br></div>Thank you very much. May be I can try to collect more logs with a modified kernel package. <br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/12 Brian Foster <span dir="ltr"><<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 04/11/2013 08:45 PM, ·ûÓÀÌÎ wrote:<br>
> the workload is about:<br>
> 24 servers, replica(3) which means the distribute is 8<br>
> load is about 3(TB)-8(TB) per day.<br>
><br>
<br>
</div>This describes your cluster, but not the workload (though cluster info<br>
is good too). What kind of workload is running on your clients (i.e.,<br>
rsync jobs, etc.)? Are you running through native gluster mount points,<br>
NFS mounts or a mix? Do you have any gluster internal operations running<br>
(i.e., rebalance, etc.).<br>
<br>
Is there any kind of pattern you can discern from the workload and when<br>
the XFS error happens to occur? You have a good number of servers in<br>
play here, is there any kind of pattern in terms of which servers<br>
experience the error? Is it always the same servers or a random set?<br>
<br>
Brian<br>
<br>
><br>
> 2013/4/12 Brian Foster <<a href="mailto:bfoster@redhat.com">bfoster@redhat.com</a> <mailto:<a href="mailto:bfoster@redhat.com">bfoster@redhat.com</a>>><br>
<div><div class="h5">><br>
> On 04/11/2013 03:11 PM, ·ûÓÀÌÎ wrote:<br>
> > It happens tonight again on one of our servers, how to debug the root<br>
> > cause? Thank you.<br>
> ><br>
><br>
> Hi,<br>
><br>
> I've attached a system tap script (stap -v xfs.stp) that should<br>
> hopefully print out a bit more data should the issue happen again. Do<br>
> you have a small enough number of nodes (or predictable enough pattern)<br>
> that you could run this on the nodes that tend to fail and collect the<br>
> output?<br>
><br>
> Also, could you collect an xfs_metadump of the filesystem in question<br>
> and make it available for download and analysis somewhere? I believe the<br>
> ideal approach is to mount/umount the filesystem first to replay the log<br>
> before collecting a metadump, but somebody could correct me on that (to<br>
> be safe, you could collect multiple dumps: pre-mount and post-mount).<br>
><br>
> Could you also describe your workload a little bit? Thanks.<br>
><br>
> Brian<br>
><br>
> > Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_iunlink_remove:<br>
> > xfs_inotobp() returned error 22.<br>
> > Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree<br>
> returned<br>
> > error 22<br>
> > Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_do_force_shutdown(0x1)<br>
> > called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address =<br>
> > 0xffffffffa02ee20a<br>
> > Apr 12 02:32:10 cqdx kernel: XFS (sdb): I/O Error Detected. Shutting<br>
> > down filesystem<br>
> > Apr 12 02:32:10 cqdx kernel: XFS (sdb): Please umount the<br>
> filesystem and<br>
> > rectify the problem(s)<br>
> > Apr 12 02:32:19 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> returned.<br>
> > Apr 12 02:32:49 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> returned.<br>
> > Apr 12 02:33:19 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> returned.<br>
> > Apr 12 02:33:49 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> returned.<br>
> ><br>
> > xfs_repair -n<br>
> ><br>
> ><br>
> > Phase 7 - verify link counts...<br>
> > would have reset inode 20021 nlinks from 0 to 1<br>
> > would have reset inode 20789 nlinks from 0 to 1<br>
> > would have reset inode 35125 nlinks from 0 to 1<br>
> > would have reset inode 35637 nlinks from 0 to 1<br>
> > would have reset inode 36149 nlinks from 0 to 1<br>
> > would have reset inode 38197 nlinks from 0 to 1<br>
> > would have reset inode 39477 nlinks from 0 to 1<br>
> > would have reset inode 54069 nlinks from 0 to 1<br>
> > would have reset inode 62261 nlinks from 0 to 1<br>
> > would have reset inode 63029 nlinks from 0 to 1<br>
> > would have reset inode 72501 nlinks from 0 to 1<br>
> > would have reset inode 79925 nlinks from 0 to 1<br>
> > would have reset inode 81205 nlinks from 0 to 1<br>
> > would have reset inode 84789 nlinks from 0 to 1<br>
> > would have reset inode 87861 nlinks from 0 to 1<br>
> > would have reset inode 90663 nlinks from 0 to 1<br>
> > would have reset inode 91189 nlinks from 0 to 1<br>
> > would have reset inode 95541 nlinks from 0 to 1<br>
> > would have reset inode 98101 nlinks from 0 to 1<br>
> > would have reset inode 101173 nlinks from 0 to 1<br>
> > would have reset inode 113205 nlinks from 0 to 1<br>
> > would have reset inode 114741 nlinks from 0 to 1<br>
> > would have reset inode 126261 nlinks from 0 to 1<br>
> > would have reset inode 140597 nlinks from 0 to 1<br>
> > would have reset inode 144693 nlinks from 0 to 1<br>
> > would have reset inode 147765 nlinks from 0 to 1<br>
> > would have reset inode 152885 nlinks from 0 to 1<br>
> > would have reset inode 161333 nlinks from 0 to 1<br>
> > would have reset inode 161845 nlinks from 0 to 1<br>
> > would have reset inode 167477 nlinks from 0 to 1<br>
> > would have reset inode 172341 nlinks from 0 to 1<br>
> > would have reset inode 191797 nlinks from 0 to 1<br>
> > would have reset inode 204853 nlinks from 0 to 1<br>
> > would have reset inode 205365 nlinks from 0 to 1<br>
> > would have reset inode 215349 nlinks from 0 to 1<br>
> > would have reset inode 215861 nlinks from 0 to 1<br>
> > would have reset inode 216373 nlinks from 0 to 1<br>
> > would have reset inode 217397 nlinks from 0 to 1<br>
> > would have reset inode 224309 nlinks from 0 to 1<br>
> > would have reset inode 225589 nlinks from 0 to 1<br>
> > would have reset inode 234549 nlinks from 0 to 1<br>
> > would have reset inode 234805 nlinks from 0 to 1<br>
> > would have reset inode 249653 nlinks from 0 to 1<br>
> > would have reset inode 250677 nlinks from 0 to 1<br>
> > would have reset inode 252469 nlinks from 0 to 1<br>
> > would have reset inode 261429 nlinks from 0 to 1<br>
> > would have reset inode 265013 nlinks from 0 to 1<br>
> > would have reset inode 266805 nlinks from 0 to 1<br>
> > would have reset inode 267317 nlinks from 0 to 1<br>
> > would have reset inode 268853 nlinks from 0 to 1<br>
> > would have reset inode 272437 nlinks from 0 to 1<br>
> > would have reset inode 273205 nlinks from 0 to 1<br>
> > would have reset inode 274229 nlinks from 0 to 1<br>
> > would have reset inode 278325 nlinks from 0 to 1<br>
> > would have reset inode 278837 nlinks from 0 to 1<br>
> > would have reset inode 281397 nlinks from 0 to 1<br>
> > would have reset inode 292661 nlinks from 0 to 1<br>
> > would have reset inode 300853 nlinks from 0 to 1<br>
> > would have reset inode 302901 nlinks from 0 to 1<br>
> > would have reset inode 305205 nlinks from 0 to 1<br>
> > would have reset inode 314165 nlinks from 0 to 1<br>
> > would have reset inode 315189 nlinks from 0 to 1<br>
> > would have reset inode 320309 nlinks from 0 to 1<br>
> > would have reset inode 324917 nlinks from 0 to 1<br>
> > would have reset inode 328245 nlinks from 0 to 1<br>
> > would have reset inode 335925 nlinks from 0 to 1<br>
> > would have reset inode 339253 nlinks from 0 to 1<br>
> > would have reset inode 339765 nlinks from 0 to 1<br>
> > would have reset inode 348213 nlinks from 0 to 1<br>
> > would have reset inode 360501 nlinks from 0 to 1<br>
> > would have reset inode 362037 nlinks from 0 to 1<br>
> > would have reset inode 366389 nlinks from 0 to 1<br>
> > would have reset inode 385845 nlinks from 0 to 1<br>
> > would have reset inode 390709 nlinks from 0 to 1<br>
> > would have reset inode 409141 nlinks from 0 to 1<br>
> > would have reset inode 413237 nlinks from 0 to 1<br>
> > would have reset inode 414773 nlinks from 0 to 1<br>
> > would have reset inode 417845 nlinks from 0 to 1<br>
> > would have reset inode 436021 nlinks from 0 to 1<br>
> > would have reset inode 439349 nlinks from 0 to 1<br>
> > would have reset inode 447029 nlinks from 0 to 1<br>
> > would have reset inode 491317 nlinks from 0 to 1<br>
> > would have reset inode 494133 nlinks from 0 to 1<br>
> > would have reset inode 495413 nlinks from 0 to 1<br>
> > would have reset inode 501301 nlinks from 0 to 1<br>
> > would have reset inode 506421 nlinks from 0 to 1<br>
> > would have reset inode 508469 nlinks from 0 to 1<br>
> > would have reset inode 508981 nlinks from 0 to 1<br>
> > would have reset inode 511797 nlinks from 0 to 1<br>
> > would have reset inode 513077 nlinks from 0 to 1<br>
> > would have reset inode 517941 nlinks from 0 to 1<br>
> > would have reset inode 521013 nlinks from 0 to 1<br>
> > would have reset inode 522805 nlinks from 0 to 1<br>
> > would have reset inode 523317 nlinks from 0 to 1<br>
> > would have reset inode 525621 nlinks from 0 to 1<br>
> > would have reset inode 527925 nlinks from 0 to 1<br>
> > would have reset inode 535605 nlinks from 0 to 1<br>
> > would have reset inode 541749 nlinks from 0 to 1<br>
> > would have reset inode 573493 nlinks from 0 to 1<br>
> > would have reset inode 578613 nlinks from 0 to 1<br>
> > would have reset inode 583029 nlinks from 0 to 1<br>
> > would have reset inode 585525 nlinks from 0 to 1<br>
> > would have reset inode 586293 nlinks from 0 to 1<br>
> > would have reset inode 586805 nlinks from 0 to 1<br>
> > would have reset inode 591413 nlinks from 0 to 1<br>
> > would have reset inode 594485 nlinks from 0 to 1<br>
> > would have reset inode 596277 nlinks from 0 to 1<br>
> > would have reset inode 603189 nlinks from 0 to 1<br>
> > would have reset inode 613429 nlinks from 0 to 1<br>
> > would have reset inode 617781 nlinks from 0 to 1<br>
> > would have reset inode 621877 nlinks from 0 to 1<br>
> > would have reset inode 623925 nlinks from 0 to 1<br>
> > would have reset inode 625205 nlinks from 0 to 1<br>
> > would have reset inode 626741 nlinks from 0 to 1<br>
> > would have reset inode 639541 nlinks from 0 to 1<br>
> > would have reset inode 640053 nlinks from 0 to 1<br>
> > would have reset inode 640565 nlinks from 0 to 1<br>
> > would have reset inode 645173 nlinks from 0 to 1<br>
> > would have reset inode 652853 nlinks from 0 to 1<br>
> > would have reset inode 656181 nlinks from 0 to 1<br>
> > would have reset inode 659253 nlinks from 0 to 1<br>
> > would have reset inode 663605 nlinks from 0 to 1<br>
> > would have reset inode 667445 nlinks from 0 to 1<br>
> > would have reset inode 680757 nlinks from 0 to 1<br>
> > would have reset inode 691253 nlinks from 0 to 1<br>
> > would have reset inode 691765 nlinks from 0 to 1<br>
> > would have reset inode 697653 nlinks from 0 to 1<br>
> > would have reset inode 700469 nlinks from 0 to 1<br>
> > would have reset inode 707893 nlinks from 0 to 1<br>
> > would have reset inode 716853 nlinks from 0 to 1<br>
> > would have reset inode 722229 nlinks from 0 to 1<br>
> > would have reset inode 722741 nlinks from 0 to 1<br>
> > would have reset inode 723765 nlinks from 0 to 1<br>
> > would have reset inode 731957 nlinks from 0 to 1<br>
> > would have reset inode 742965 nlinks from 0 to 1<br>
> > would have reset inode 743477 nlinks from 0 to 1<br>
> > would have reset inode 745781 nlinks from 0 to 1<br>
> > would have reset inode 746293 nlinks from 0 to 1<br>
> > would have reset inode 774453 nlinks from 0 to 1<br>
> > would have reset inode 778805 nlinks from 0 to 1<br>
> > would have reset inode 785013 nlinks from 0 to 1<br>
> > would have reset inode 785973 nlinks from 0 to 1<br>
> > would have reset inode 791349 nlinks from 0 to 1<br>
> > would have reset inode 796981 nlinks from 0 to 1<br>
> > would have reset inode 803381 nlinks from 0 to 1<br>
> > would have reset inode 806965 nlinks from 0 to 1<br>
> > would have reset inode 811798 nlinks from 0 to 1<br>
> > would have reset inode 812310 nlinks from 0 to 1<br>
> > would have reset inode 813078 nlinks from 0 to 1<br>
> > would have reset inode 813607 nlinks from 0 to 1<br>
> > would have reset inode 814183 nlinks from 0 to 1<br>
> > would have reset inode 822069 nlinks from 0 to 1<br>
> > would have reset inode 828469 nlinks from 0 to 1<br>
> > would have reset inode 830005 nlinks from 0 to 1<br>
> > would have reset inode 832053 nlinks from 0 to 1<br>
> > would have reset inode 832565 nlinks from 0 to 1<br>
> > would have reset inode 836661 nlinks from 0 to 1<br>
> > would have reset inode 841013 nlinks from 0 to 1<br>
> > would have reset inode 841525 nlinks from 0 to 1<br>
> > would have reset inode 845365 nlinks from 0 to 1<br>
> > would have reset inode 846133 nlinks from 0 to 1<br>
> > would have reset inode 847157 nlinks from 0 to 1<br>
> > would have reset inode 852533 nlinks from 0 to 1<br>
> > would have reset inode 857141 nlinks from 0 to 1<br>
> > would have reset inode 863271 nlinks from 0 to 1<br>
> > would have reset inode 866855 nlinks from 0 to 1<br>
> > would have reset inode 887861 nlinks from 0 to 1<br>
> > would have reset inode 891701 nlinks from 0 to 1<br>
> > would have reset inode 894773 nlinks from 0 to 1<br>
> > would have reset inode 900149 nlinks from 0 to 1<br>
> > would have reset inode 902197 nlinks from 0 to 1<br>
> > would have reset inode 906293 nlinks from 0 to 1<br>
> > would have reset inode 906805 nlinks from 0 to 1<br>
> > would have reset inode 909877 nlinks from 0 to 1<br>
> > would have reset inode 925493 nlinks from 0 to 1<br>
> > would have reset inode 949543 nlinks from 0 to 1<br>
> > would have reset inode 955175 nlinks from 0 to 1<br>
> > would have reset inode 963623 nlinks from 0 to 1<br>
> > would have reset inode 967733 nlinks from 0 to 1<br>
> > would have reset inode 968231 nlinks from 0 to 1<br>
> > would have reset inode 982069 nlinks from 0 to 1<br>
> > would have reset inode 1007413 nlinks from 0 to 1<br>
> > would have reset inode 1011509 nlinks from 0 to 1<br>
> > would have reset inode 1014069 nlinks from 0 to 1<br>
> > would have reset inode 1014581 nlinks from 0 to 1<br>
> > would have reset inode 1022005 nlinks from 0 to 1<br>
> > would have reset inode 1022517 nlinks from 0 to 1<br>
> > would have reset inode 1023029 nlinks from 0 to 1<br>
> > would have reset inode 1025333 nlinks from 0 to 1<br>
> > would have reset inode 1043765 nlinks from 0 to 1<br>
> > would have reset inode 1044789 nlinks from 0 to 1<br>
> > would have reset inode 1049397 nlinks from 0 to 1<br>
> > would have reset inode 1050933 nlinks from 0 to 1<br>
> > would have reset inode 1051445 nlinks from 0 to 1<br>
> > would have reset inode 1054261 nlinks from 0 to 1<br>
> > would have reset inode 1060917 nlinks from 0 to 1<br>
> > would have reset inode 1063477 nlinks from 0 to 1<br>
> > would have reset inode 1076021 nlinks from 0 to 1<br>
> > would have reset inode 1081141 nlinks from 0 to 1<br>
> > would have reset inode 1086261 nlinks from 0 to 1<br>
> > would have reset inode 1097269 nlinks from 0 to 1<br>
> > would have reset inode 1099829 nlinks from 0 to 1<br>
> > would have reset inode 1100853 nlinks from 0 to 1<br>
> > would have reset inode 1101877 nlinks from 0 to 1<br>
> > would have reset inode 1126709 nlinks from 0 to 1<br>
> > would have reset inode 1134389 nlinks from 0 to 1<br>
> > would have reset inode 1141045 nlinks from 0 to 1<br>
> > would have reset inode 1141557 nlinks from 0 to 1<br>
> > would have reset inode 1142581 nlinks from 0 to 1<br>
> > would have reset inode 1148469 nlinks from 0 to 1<br>
> > would have reset inode 1153333 nlinks from 0 to 1<br>
> > would have reset inode 1181749 nlinks from 0 to 1<br>
> > would have reset inode 1192245 nlinks from 0 to 1<br>
> > would have reset inode 1198133 nlinks from 0 to 1<br>
> > would have reset inode 1203765 nlinks from 0 to 1<br>
> > would have reset inode 1221429 nlinks from 0 to 1<br>
> > would have reset inode 1223989 nlinks from 0 to 1<br>
> > would have reset inode 1235509 nlinks from 0 to 1<br>
> > would have reset inode 1239349 nlinks from 0 to 1<br>
> > would have reset inode 1240885 nlinks from 0 to 1<br>
> > would have reset inode 1241397 nlinks from 0 to 1<br>
> > would have reset inode 1241909 nlinks from 0 to 1<br>
> > would have reset inode 1242421 nlinks from 0 to 1<br>
> > would have reset inode 1244981 nlinks from 0 to 1<br>
> > would have reset inode 1246517 nlinks from 0 to 1<br>
> > would have reset inode 1253429 nlinks from 0 to 1<br>
> > would have reset inode 1271861 nlinks from 0 to 1<br>
> > would have reset inode 1274677 nlinks from 0 to 1<br>
> > would have reset inode 1277749 nlinks from 0 to 1<br>
> > would have reset inode 1278773 nlinks from 0 to 1<br>
> > would have reset inode 1286709 nlinks from 0 to 1<br>
> > would have reset inode 1288245 nlinks from 0 to 1<br>
> > would have reset inode 1299765 nlinks from 0 to 1<br>
> > would have reset inode 1302325 nlinks from 0 to 1<br>
> > would have reset inode 1304885 nlinks from 0 to 1<br>
> > would have reset inode 1305397 nlinks from 0 to 1<br>
> > would have reset inode 1307509 nlinks from 0 to 1<br>
> > would have reset inode 1309493 nlinks from 0 to 1<br>
> > would have reset inode 1310517 nlinks from 0 to 1<br>
> > would have reset inode 1311029 nlinks from 0 to 1<br>
> > would have reset inode 1312053 nlinks from 0 to 1<br>
> > would have reset inode 1316917 nlinks from 0 to 1<br>
> > would have reset inode 1317941 nlinks from 0 to 1<br>
> > would have reset inode 1320821 nlinks from 0 to 1<br>
> > would have reset inode 1322805 nlinks from 0 to 1<br>
> > would have reset inode 1332789 nlinks from 0 to 1<br>
> > would have reset inode 1336373 nlinks from 0 to 1<br>
> > would have reset inode 1345653 nlinks from 0 to 1<br>
> > would have reset inode 1354549 nlinks from 0 to 1<br>
> > would have reset inode 1361973 nlinks from 0 to 1<br>
> > would have reset inode 1369909 nlinks from 0 to 1<br>
> > would have reset inode 1372981 nlinks from 0 to 1<br>
> > would have reset inode 1388853 nlinks from 0 to 1<br>
> > would have reset inode 1402933 nlinks from 0 to 1<br>
> > would have reset inode 1403445 nlinks from 0 to 1<br>
> > would have reset inode 1420085 nlinks from 0 to 1<br>
> > would have reset inode 1452853 nlinks from 0 to 1<br>
> > would have reset inode 1456437 nlinks from 0 to 1<br>
> > would have reset inode 1457973 nlinks from 0 to 1<br>
> > would have reset inode 1459253 nlinks from 0 to 1<br>
> > would have reset inode 1467957 nlinks from 0 to 1<br>
> > would have reset inode 1471541 nlinks from 0 to 1<br>
> > would have reset inode 1476661 nlinks from 0 to 1<br>
> > would have reset inode 1479733 nlinks from 0 to 1<br>
> > would have reset inode 1483061 nlinks from 0 to 1<br>
> > would have reset inode 1484085 nlinks from 0 to 1<br>
> > would have reset inode 1486133 nlinks from 0 to 1<br>
> > would have reset inode 1489461 nlinks from 0 to 1<br>
> > would have reset inode 1490037 nlinks from 0 to 1<br>
> > would have reset inode 1492021 nlinks from 0 to 1<br>
> > would have reset inode 1493557 nlinks from 0 to 1<br>
> > would have reset inode 1494069 nlinks from 0 to 1<br>
> > would have reset inode 1496885 nlinks from 0 to 1<br>
> > would have reset inode 1498421 nlinks from 0 to 1<br>
> > would have reset inode 1498933 nlinks from 0 to 1<br>
> > would have reset inode 1499957 nlinks from 0 to 1<br>
> > would have reset inode 1506101 nlinks from 0 to 1<br>
> > would have reset inode 1507637 nlinks from 0 to 1<br>
> > would have reset inode 1510453 nlinks from 0 to 1<br>
> > would have reset inode 1514293 nlinks from 0 to 1<br>
> > would have reset inode 1517365 nlinks from 0 to 1<br>
> > would have reset inode 1520693 nlinks from 0 to 1<br>
> > would have reset inode 1521973 nlinks from 0 to 1<br>
> > would have reset inode 1530421 nlinks from 0 to 1<br>
> > would have reset inode 1530933 nlinks from 0 to 1<br>
> > would have reset inode 1537333 nlinks from 0 to 1<br>
> > would have reset inode 1538357 nlinks from 0 to 1<br>
> > would have reset inode 1548853 nlinks from 0 to 1<br>
> > would have reset inode 1553973 nlinks from 0 to 1<br>
> > would have reset inode 1557301 nlinks from 0 to 1<br>
> > would have reset inode 1564213 nlinks from 0 to 1<br>
> > would have reset inode 1564725 nlinks from 0 to 1<br>
> > would have reset inode 1576501 nlinks from 0 to 1<br>
> > would have reset inode 1580597 nlinks from 0 to 1<br>
> > would have reset inode 1584693 nlinks from 0 to 1<br>
> > would have reset inode 1586485 nlinks from 0 to 1<br>
> > would have reset inode 1589301 nlinks from 0 to 1<br>
> > would have reset inode 1589813 nlinks from 0 to 1<br>
> > would have reset inode 1592629 nlinks from 0 to 1<br>
> > would have reset inode 1595701 nlinks from 0 to 1<br>
> > would have reset inode 1601077 nlinks from 0 to 1<br>
> > would have reset inode 1623861 nlinks from 0 to 1<br>
> > would have reset inode 1626677 nlinks from 0 to 1<br>
> > would have reset inode 1627701 nlinks from 0 to 1<br>
> > would have reset inode 1633333 nlinks from 0 to 1<br>
> > would have reset inode 1639221 nlinks from 0 to 1<br>
> > would have reset inode 1649205 nlinks from 0 to 1<br>
> > would have reset inode 1686325 nlinks from 0 to 1<br>
> > would have reset inode 1690677 nlinks from 0 to 1<br>
> > would have reset inode 1693749 nlinks from 0 to 1<br>
> > would have reset inode 1704757 nlinks from 0 to 1<br>
> > would have reset inode 1707061 nlinks from 0 to 1<br>
> > would have reset inode 1709109 nlinks from 0 to 1<br>
> > would have reset inode 1719349 nlinks from 0 to 1<br>
> > would have reset inode 1737013 nlinks from 0 to 1<br>
> > would have reset inode 1741365 nlinks from 0 to 1<br>
> > would have reset inode 1747509 nlinks from 0 to 1<br>
> > would have reset inode 1770805 nlinks from 0 to 1<br>
> > would have reset inode 1780789 nlinks from 0 to 1<br>
> > would have reset inode 1793589 nlinks from 0 to 1<br>
> > would have reset inode 1795125 nlinks from 0 to 1<br>
> > would have reset inode 1800757 nlinks from 0 to 1<br>
> > would have reset inode 1801269 nlinks from 0 to 1<br>
> > would have reset inode 1802549 nlinks from 0 to 1<br>
> > would have reset inode 1804085 nlinks from 0 to 1<br>
> > would have reset inode 1817141 nlinks from 0 to 1<br>
> > would have reset inode 1821749 nlinks from 0 to 1<br>
> > would have reset inode 1832757 nlinks from 0 to 1<br>
> > would have reset inode 1836341 nlinks from 0 to 1<br>
> > would have reset inode 1856309 nlinks from 0 to 1<br>
> > would have reset inode 1900597 nlinks from 0 to 1<br>
> > would have reset inode 1902901 nlinks from 0 to 1<br>
> > would have reset inode 1912373 nlinks from 0 to 1<br>
> > would have reset inode 1943093 nlinks from 0 to 1<br>
> > would have reset inode 1944373 nlinks from 0 to 1<br>
> > would have reset inode 1954101 nlinks from 0 to 1<br>
> > would have reset inode 1955893 nlinks from 0 to 1<br>
> > would have reset inode 1961781 nlinks from 0 to 1<br>
> > would have reset inode 1974325 nlinks from 0 to 1<br>
> > would have reset inode 1978677 nlinks from 0 to 1<br>
> > would have reset inode 1981237 nlinks from 0 to 1<br>
> > would have reset inode 1992245 nlinks from 0 to 1<br>
> > would have reset inode 2000949 nlinks from 0 to 1<br>
> > would have reset inode 2002229 nlinks from 0 to 1<br>
> > would have reset inode 2004789 nlinks from 0 to 1<br>
> > would have reset inode 2005301 nlinks from 0 to 1<br>
> > would have reset inode 2011189 nlinks from 0 to 1<br>
> > would have reset inode 2012981 nlinks from 0 to 1<br>
> > would have reset inode 2015285 nlinks from 0 to 1<br>
> > would have reset inode 2018869 nlinks from 0 to 1<br>
> > would have reset inode 2028341 nlinks from 0 to 1<br>
> > would have reset inode 2028853 nlinks from 0 to 1<br>
> > would have reset inode 2030901 nlinks from 0 to 1<br>
> > would have reset inode 2032181 nlinks from 0 to 1<br>
> > would have reset inode 2032693 nlinks from 0 to 1<br>
> > would have reset inode 2040117 nlinks from 0 to 1<br>
> > would have reset inode 2053685 nlinks from 0 to 1<br>
> > would have reset inode 2083893 nlinks from 0 to 1<br>
> > would have reset inode 2087221 nlinks from 0 to 1<br>
> > would have reset inode 2095925 nlinks from 0 to 1<br>
> > would have reset inode 2098741 nlinks from 0 to 1<br>
> > would have reset inode 2100533 nlinks from 0 to 1<br>
> > would have reset inode 2101301 nlinks from 0 to 1<br>
> > would have reset inode 2123573 nlinks from 0 to 1<br>
> > would have reset inode 2132789 nlinks from 0 to 1<br>
> > would have reset inode 2133813 nlinks from 0 to 1<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > 2013/4/10 ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a>><br>
</div></div>> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a>>>><br>
<div><div class="h5">> ><br>
> > The storage info is as following:<br>
> > RAID-6<br>
> > SATA HDD<br>
> > Controller: PERC H710P Mini (Embedded)<br>
> > Disk /dev/sdb: 30000.3 GB, 30000346562560 bytes<br>
> > 255 heads, 63 sectors/track, 3647334 cylinders<br>
> > Units = cylinders of 16065 * 512 = 8225280 bytes<br>
> > Sector size (logical/physical): 512 bytes / 512 bytes<br>
> > I/O size (minimum/optimal): 512 bytes / 512 bytes<br>
> > Disk identifier: 0x00000000<br>
> ><br>
> > sd 0:2:1:0: [sdb] 58594426880 512-byte logical blocks: (30.0<br>
> TB/27.2<br>
> > TiB)<br>
> > sd 0:2:1:0: [sdb] Write Protect is off<br>
> > sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08<br>
> > sd 0:2:1:0: [sdb] Write cache: enabled, read cache: enabled,<br>
> doesn't<br>
> > support DPO or FUA<br>
> > sd 0:2:1:0: [sdb] Attached SCSI disk<br>
> ><br>
> > *-storage<br>
> > description: RAID bus controller<br>
> > product: MegaRAID SAS 2208 [Thunderbolt]<br>
> > vendor: LSI Logic / Symbios Logic<br>
> > physical id: 0<br>
> > bus info: pci@0000:02:00.0<br>
> > logical name: scsi0<br>
> > version: 01<br>
> > width: 64 bits<br>
> > clock: 33MHz<br>
> > capabilities: storage pm pciexpress vpd msi msix bus_master<br>
> > cap_list rom<br>
> > configuration: driver=megaraid_sas latency=0<br>
> > resources: irq:42 ioport:fc00(size=256)<br>
> > memory:dd7fc000-dd7fffff memory:dd780000-dd7bffff<br>
> > memory:dc800000-dc81ffff(prefetchable)<br>
> > *-disk:0<br>
> > description: SCSI Disk<br>
> > product: PERC H710P<br>
> > vendor: DELL<br>
> > physical id: 2.0.0<br>
> > bus info: scsi@0:2.0.0<br>
> > logical name: /dev/sda<br>
> > version: 3.13<br>
> > serial: 0049d6ce1d9f2035180096fde490f648<br>
> > size: 558GiB (599GB)<br>
> > capabilities: partitioned partitioned:dos<br>
> > configuration: ansiversion=5 signature=000aa336<br>
> > *-disk:1<br>
> > description: SCSI Disk<br>
> > product: PERC H710P<br>
> > vendor: DELL<br>
> > physical id: 2.1.0<br>
> > bus info: scsi@0:2.1.0<br>
> > logical name: /dev/sdb<br>
> > logical name: /mnt/xfsd<br>
> > version: 3.13<br>
> > serial: 003366f71da22035180096fde490f648<br>
> > size: 27TiB (30TB)<br>
> > configuration: ansiversion=5 mount.fstype=xfs<br>
> ><br>
> mount.options=rw,relatime,attr2,delaylog,logbsize=64k,sunit=128,swidth=1280,noquota<br>
> > state=mounted<br>
> ><br>
> > Thank you.<br>
> ><br>
> ><br>
> > 2013/4/10 Emmanuel Florac <<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a><br>
> <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a>><br>
</div></div>> > <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a> <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a>>>><br>
<div class="im">> ><br>
> > Le Tue, 9 Apr 2013 23:10:03 +0800<br>
> > ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a>><br>
</div>> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com">yongtaofu@gmail.com</a>>>> ¨¦crivait:<br>
<div class="im">> ><br>
> > > > Apr 9 11:01:30 cqdx kernel: XFS (sdb): I/O Error<br>
> Detected.<br>
> > > > Shutting down filesystem<br>
> ><br>
> > This. I/O error detected. That means that at some point the<br>
> > underlying<br>
> > device (disk, RAID array, SAN volume) couldn't be reached.<br>
> So this<br>
> > could very well be a case of a flakey drive, array, cable<br>
> or SCSI<br>
> > driver.<br>
> ><br>
> > What's the storage setup here?<br>
> ><br>
> > --<br>
> ><br>
> ------------------------------------------------------------------------<br>
> > Emmanuel Florac | Direction technique<br>
> > | Intellique<br>
> > | <<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a><br>
> <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a>><br>
</div>> > <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a><br>
<div class="im">> <mailto:<a href="mailto:eflorac@intellique.com">eflorac@intellique.com</a>>>><br>
> > | +33 1 78 94 84 02<br>
</div>> <tel:%2B33%201%2078%2094%2084%2002><br>
<div class="im">> ><br>
> ------------------------------------------------------------------------<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > ·ûÓÀÌÎ<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > ·ûÓÀÌÎ<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > xfs mailing list<br>
</div>> > <a href="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a> <mailto:<a href="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a>><br>
<div class="HOEnZb"><div class="h5">> > <a href="http://oss.sgi.com/mailman/listinfo/xfs" target="_blank">http://oss.sgi.com/mailman/listinfo/xfs</a><br>
> ><br>
><br>
><br>
><br>
><br>
> --<br>
> ·ûÓÀÌÎ<br>
><br>
><br>
> _______________________________________________<br>
> xfs mailing list<br>
> <a href="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a><br>
> <a href="http://oss.sgi.com/mailman/listinfo/xfs" target="_blank">http://oss.sgi.com/mailman/listinfo/xfs</a><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>