<div dir="ltr"><div><div>Hi Brain,<br></div>Here's the meta_dump file:<br><a href="https://docs.google.com/file/d/0B7n2C4T5tfNCRGpoUWIzaTlvM0E/edit?usp=sharing">https://docs.google.com/file/d/0B7n2C4T5tfNCRGpoUWIzaTlvM0E/edit?usp=sharing</a><br>
</div><div>Thank you.<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div>Hi <span name="Eric Sandeen">Eric,<br></span></div><span name="Eric Sandeen">I'm sorry for spaming. <br></span></div><span name="Eric Sandeen">And I got some more info and hope you're interested.<br>
</span></div><span name="Eric Sandeen">In glusterfs3.3<br></span></div><span name="Eric Sandeen">glusterfsd/src/glusterfsd.c line 1332 there's an unlink operation.<br> if (ctx->cmd_args.pid_file) {<br>
unlink (ctx->cmd_args.pid_file);<br> ctx->cmd_args.pid_file = NULL;<br> }<br></span></div><span name="Eric Sandeen">Glusterfs try to unlink the rebalance pid file after complete and may be this is where the issue happens.<br>
</span></div><span name="Eric Sandeen">See logs bellow:<br>1.<br></span></div><span name="Eric Sandeen">/var/log/secure indicates I start rebalance on Apr 15 11:58:11<br>Apr 15 11:58:11 10 sudo: root : TTY=pts/2 ; PWD=/root ; USER=root ; COMMAND=/usr/sbin/gluster volume rebalance testbug start<br>
2.<br></span></div><span name="Eric Sandeen">After xfs shutdown I got the following log:<div class="im"><br>--- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1680").return -- return=0x16<br>
vars: tp=0xffff881c81797c70 ip=0xffff881003c13c00 next_ino=? mp=? agi=? dip=? agibp=0xffff880109b47e20 ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff882000000000 bucket_index=? offset=? last_offset=0xffffffffffff8810 error=? __func__=[...]<br>
ip: i_ino = 0x113, i_flags = 0x0<br></div></span></div><span name="Eric Sandeen">the inode is </span>lead to xfs shutdown is<br><span name="Eric Sandeen">0x113<br>3. <br></span></div><span name="Eric Sandeen">I repair xfs and in lost+foud I find the inode:<br>
[<a href="mailto:root@10.23.72.93" target="_blank">root@10.23.72.93</a> lost+found]# pwd<br>/mnt/xfsd/lost+found<br>[<a href="mailto:root@10.23.72.93" target="_blank">root@10.23.72.93</a> lost+found]# ls -l 275<br>---------T 1 root root 0 Apr 15 11:58 275<br>
[<a href="mailto:root@10.23.72.93" target="_blank">root@10.23.72.93</a> lost+found]# stat 275<br> File: `275'<br> Size: 0 Blocks: 0 IO Block: 4096 regular empty file<br>Device: 810h/2064d Inode: 275 Links: 1<br>
Access: (1000/---------T) Uid: ( 0/ root) Gid: ( 0/ root)<br>Access: 2013-04-15 11:58:25.833443445 +0800<br>Modify: 2013-04-15 11:58:25.912461256 +0800<br>Change: 2013-04-15 11:58:25.915442091 +0800<br></span></div>
<div><span name="Eric Sandeen">This file is created aroud </span><span name="Eric Sandeen">2013-04-15 11:58.</span></div><span name="Eric Sandeen">And the other files in lost+foud has extended attribute but this file doesn't. Which means it is not part of glusterfs backend files. It should be the rebalance pid file.<br>
<br></span></div><span name="Eric Sandeen">So may be unlink the rebalance pid file leads to xfs shutdown.<br><br></span></div><span name="Eric Sandeen">Thank you.<br></span><div><div><div><div><div><div>
<span name="Eric Sandeen"><br></span></div></div></div></div></div></div></div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">2013/4/15 Eric Sandeen <span dir="ltr"><<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 4/15/13 8:45 AM, ·ûÓÀÌÎ wrote:<br>
> And at the same time we got the following error log of glusterfs:<br>
> [2013-04-15 20:43:03.851163] I [dht-rebalance.c:1611:gf_defrag_status_get] 0-glusterfs: Rebalance is completed<br>
> [2013-04-15 20:43:03.851248] I [dht-rebalance.c:1614:gf_defrag_status_get] 0-glusterfs: Files migrated: 1629, size: 1582329065954, lookups: 11036, failures: 561<br>
> [2013-04-15 20:43:03.887634] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3bd16e767d] (-->/lib64/libpthread.so.0() [0x3bd1a07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405c9d]))) 0-: received signum (15), shutting down<br>
> [2013-04-15 20:43:03.887878] E [rpcsvc.c:1155:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap<br>
><br>
<br>
</div>We'll take a look, thanks.<br>
<br>
Going forward, could I ask that you take a few minutes to batch up the information, rather than sending several emails in a row? It makes it much harder to collect the information when it's spread across so many emails.<br>
<br>
Thanks,<br>
-Eric<br>
<br>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>