<div dir="ltr">And at the same time we got the following error log of glusterfs:<br>[2013-04-15 20:43:03.851163] I [dht-rebalance.c:1611:gf_defrag_status_get] 0-glusterfs: Rebalance is completed<br>[2013-04-15 20:43:03.851248] I [dht-rebalance.c:1614:gf_defrag_status_get] 0-glusterfs: Files migrated: 1629, size: 1582329065954, lookups: 11036, failures: 561<br>
[2013-04-15 20:43:03.887634] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3bd16e767d] (-->/lib64/libpthread.so.0() [0x3bd1a07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405c9d]))) 0-: received signum (15), shutting down<br>
[2013-04-15 20:43:03.887878] E [rpcsvc.c:1155:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">More info about it is: It happened exactly when glusterfs rebalance complete.<br></div><div class="gmail_extra">
<div><div class="h5"><br><br><div class="gmail_quote">2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">and xfs kernel trace is:<br>Apr 15 20:43:03 10 kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22.<br>
Apr 15 20:43:03 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22<br>Apr 15 20:43:03 10 kernel: Pid: 3093, comm: glusterfsd Not tainted 2.6.32-279.el6.x86_64 #1<br>
Apr 15 20:43:03 10 kernel: Call Trace:<br>Apr 15 20:43:03 10 kernel: [<ffffffffa02d4212>] ? xfs_inactive+0x442/0x460 [xfs]<br>Apr 15 20:43:03 10 kernel: [<ffffffffa02e1790>] ? xfs_fs_clear_inode+0xa0/0xd0 [xfs]<br>
Apr 15 20:43:03 10 kernel: [<ffffffff81195adc>] ? clear_inode+0xac/0x140<br>Apr 15 20:43:03 10 kernel: [<ffffffff81196296>] ? generic_delete_inode+0x196/0x1d0<br>Apr 15 20:43:03 10 kernel: [<ffffffff81196335>] ? generic_drop_inode+0x65/0x80<br>
Apr 15 20:43:03 10 kernel: [<ffffffff81195182>] ? iput+0x62/0x70<br>Apr 15 20:43:03 10 kernel: [<ffffffff81191ce0>] ? dentry_iput+0x90/0x100<br>Apr 15 20:43:03 10 kernel: [<ffffffff81191e41>] ? d_kill+0x31/0x60<br>
Apr 15 20:43:03 10 kernel: [<ffffffff8119386c>] ? dput+0x7c/0x150<br>Apr 15 20:43:03 10 kernel: [<ffffffff8117c9c9>] ? __fput+0x189/0x210<br>Apr 15 20:43:03 10 kernel: [<ffffffff8117ca75>] ? fput+0x25/0x30<br>
Apr 15 20:43:03 10 kernel: [<ffffffff8117849d>] ? filp_close+0x5d/0x90<br>Apr 15 20:43:03 10 kernel: [<ffffffff81178575>] ? sys_close+0xa5/0x100<br>Apr 15 20:43:03 10 kernel: [<ffffffff8100b308>] ? tracesys+0xd9/0xde<br>
Apr 15 20:43:03 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1186 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02d422b<br>Apr 15 20:43:03 10 kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem<br>
Apr 15 20:43:03 10 kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s)<br>Apr 15 20:43:13 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.<br></div><div class="gmail_extra"><div><div>
<br><br><div class="gmail_quote">
2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div>Dear Brian and xfs experts,<br></div>Brain your scripts works and I am able to reproduce it with glusterfs rebalance on our test cluster. 2 of our server xfs shutdown during glusterfs rebalance, the shutdown userspace stacktrace both related to pthread. See logs bellow, What's your opinion? Thank you very much!<br>
</div>logs:<br>[<a href="mailto:root@10.23.72.93" target="_blank">root@10.23.72.93</a> ~]# cat xfs.log <br><br>--- xfs_imap -- module("xfs").function("xfs_imap@fs/xfs/xfs_ialloc.c:1257").return -- return=0x16<br>
vars: mp=0xffff882017a50800 tp=0xffff881c81797c70 ino=0xffffffff imap=0xffff88100e2f7c08 flags=0x0 agbno=? agino=? agno=? blks_per_cluster=? chunk_agbno=? cluster_agbno=? error=? offset=? offset_agbno=? __func__=[...]<br>
mp: m_agno_log = 0x5, m_agino_log = 0x20<br>mp->m_sb: sb_agcount = 0x1c, sb_agblocks = 0xffffff0, sb_inopblog = 0x4, sb_agblklog = 0x1c, sb_dblocks = 0x1b4900000<br>imap: im_blkno = 0x0, im_len = 0xa078, im_boffset = 0x86ea<br>
kernel backtrace:<br>Returning from: 0xffffffffa02b3ab0 : xfs_imap+0x0/0x280 [xfs]<br>Returning to : 0xffffffffa02b9599 : xfs_inotobp+0x49/0xc0 [xfs]<br> 0xffffffffa02b96f1 : xfs_iunlink_remove+0xe1/0x320 [xfs]<br> 0xffffffff81501a69<br>
0x0 (inexact)<br>user backtrace:<br> 0x3bd1a0e5ad [/lib64/<a href="http://libpthread-2.12.so" target="_blank">libpthread-2.12.so</a>+0xe5ad/0x219000]<br><br>--- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1680").return -- return=0x16<br>
vars: tp=0xffff881c81797c70 ip=0xffff881003c13c00 next_ino=? mp=? agi=? dip=? agibp=0xffff880109b47e20 ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff882000000000 bucket_index=? offset=? last_offset=0xffffffffffff8810 error=? __func__=[...]<br>
ip: i_ino = 0x113, i_flags = 0x0<br>ip->i_d: di_nlink = 0x0, di_gen = 0x0<br>[<a href="mailto:root@10.23.72.93" target="_blank">root@10.23.72.93</a> ~]# <br>[<a href="mailto:root@10.23.72.94" target="_blank">root@10.23.72.94</a> ~]# cat xfs.log <br>
<br>
--- xfs_imap -- module("xfs").function("xfs_imap@fs/xfs/xfs_ialloc.c:1257").return -- return=0x16<br>vars: mp=0xffff881017c6c800 tp=0xffff8801037acea0 ino=0xffffffff imap=0xffff882017101c08 flags=0x0 agbno=? agino=? agno=? blks_per_cluster=? chunk_agbno=? cluster_agbno=? error=? offset=? offset_agbno=? __func__=[...]<br>
mp: m_agno_log = 0x5, m_agino_log = 0x20<br>mp->m_sb: sb_agcount = 0x1c, sb_agblocks = 0xffffff0, sb_inopblog = 0x4, sb_agblklog = 0x1c, sb_dblocks = 0x1b4900000<br>imap: im_blkno = 0x0, im_len = 0xd98, im_boffset = 0x547<br>
kernel backtrace:<br>Returning from: 0xffffffffa02b3ab0 : xfs_imap+0x0/0x280 [xfs]<br>Returning to : 0xffffffffa02b9599 : xfs_inotobp+0x49/0xc0 [xfs]<br> 0xffffffffa02b96f1 : xfs_iunlink_remove+0xe1/0x320 [xfs]<br> 0xffffffff81501a69<br>
0x0 (inexact)<br>user backtrace:<br> 0x30cd40e5ad [/lib64/<a href="http://libpthread-2.12.so" target="_blank">libpthread-2.12.so</a>+0xe5ad/0x219000]<br><br>--- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1680").return -- return=0x16<br>
vars: tp=0xffff8801037acea0 ip=0xffff880e697c8800 next_ino=? mp=? agi=? dip=? agibp=0xffff880d846c2d60 ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff881017c6c800 bucket_index=? offset=? last_offset=0xffffffffffff880e error=? __func__=[...]<br>
ip: i_ino = 0x142, i_flags = 0x0<br>ip->i_d: di_nlink = 0x0, di_gen = 0x3565732e<br><br></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Also glusterfs use a lot of hardlink for self-heal:<br>--------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/998416323<br>
---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/999296624<br>---------T 2 root root 0 Apr 15 12:24 /mnt/xfsd/testbug/999568484<br>
---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/999956875<br>---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/.glusterfs/05/2f/052f4e3e-c379-4a3c-b995-a10fdaca33d0<br>---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/.glusterfs/05/95/0595272e-ce2b-45d5-8693-d02c00b94d9d<br>
---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/.glusterfs/05/ca/05ca00a0-92a7-44cf-b6e3-380496aafaa4<br>---------T 2 root root 0 Apr 15 12:24 /mnt/xfsd/testbug/.glusterfs/0a/23/0a238ca7-3cef-4540-9c98-6bf631551b21<br>
---------T 2 root root 0 Apr 15 11:58 /mnt/xfsd/testbug/.glusterfs/0a/4b/0a4b640b-f675-4708-bb59-e2369ffbbb9d<br></div>Does it related?<br></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">
2013/4/15 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Dear xfs experts,<br></div>Now I'm deploying Brian's system script in out cluster. But from last night till now 5 servers in our 24 servers xfs shutdown with the same error. I run xfs_repair command and found all the lost inodes are glusterfs dht link files. This explains why the xfs shutdown tend to happen during glusterfs rebalance. During glusterfs rebalance procedure a lot of dhk link files may be unlinked. For example the following inodes are found in lost+found in one of the servers:<br>
[root@* lost+found]# pwd<br>/mnt/xfsd/lost+found<br>[root@* lost+found]# ls -l<br>total 740<br>---------T 1 root root 0 Apr 8 21:06 100119<br>---------T 1 root root 0 Apr 8 21:11 101123<br>---------T 1 root root 0 Apr 8 21:19 102659<br>
---------T 1 root root 0 Apr 12 14:46 1040919<br>---------T 1 root root 0 Apr 12 14:58 1041943<br>---------T 1 root root 0 Apr 8 21:32 105219<br>---------T 1 root root 0 Apr 8 21:37 105731<br>---------T 1 root root 0 Apr 12 17:48 1068055<br>
---------T 1 root root 0 Apr 12 18:38 1073943<br>---------T 1 root root 0 Apr 8 21:54 108035<br>---------T 1 root root 0 Apr 12 21:49 1091095<br>---------T 1 root root 0 Apr 13 00:17 1111063<br>---------T 1 root root 0 Apr 13 03:51 1121815<br>
---------T 1 root root 0 Apr 8 22:25 112387<br>---------T 1 root root 0 Apr 13 06:39 1136151<br>...<br>[root@* lost+found]# getfattr -m . -d -e hex *<br><br># file: 96007<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>
trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>trusted.gfid=0xa0370d8a9f104dafbebbd0e6dd7ce1f7<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3600<br>
trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x0000000049dff000<br><br># file: 97027<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>
trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>trusted.gfid=0xc1c1fe2ec7034442a623385f43b04c25<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3600<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x000000006ac78000<br>
<br># file: 97559<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>
trusted.gfid=0xcf7c17013c914511bda4d1c743fae118<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3500<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x00000000519fb000<br>
<br># file: 98055<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>
trusted.gfid=0xe86abc6e2c4b44c28d415fbbe34f2102<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3600<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x000000004c098000<br>
<br># file: 98567<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>
trusted.gfid=0x12543a2efbdf4b9fa61c6d89ca396f80<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3500<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x000000006bc98000<br>
<br># file: 98583<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>
trusted.gfid=0x760d16d3b7974cfb9c0a665a0982c470<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3500<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x000000006cde9000<br>
<br># file: 99607<br>trusted.afr.mams-cq-mt-video-client-3=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-4=0x000000000000000000000000<br>trusted.afr.mams-cq-mt-video-client-5=0x000000000000000000000000<br>
trusted.gfid=0x0849a732ea204bc3b8bae830b46881da<br>trusted.glusterfs.dht.linkto=0x6d616d732d63712d6d742d766964656f2d7265706c69636174652d3500<br>trusted.glusterfs.quota.ca34e1ce-f046-4ed4-bbd1-261b21bfe0b8.contri=0x00000000513f1000<br>
...<br><br></div>What do you think about it? Thank you very much.<br></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/12 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Brian,<br><br></div>Your scripts works for me now after I installed all the rpm built out from kernel srpm. I'll try it. Thank you.<br>
</div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">
2013/4/12 Brian Foster <span dir="ltr"><<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 04/12/2013 04:32 AM, ·ûÓÀÌÎ wrote:<br>
> Dear xfs experts,<br>
> Can I just call xfs_stack_trace(); in the second line of<br>
> xfs_do_force_shutdown() to print stack and rebuild kernel to check<br>
> what's the error?<br>
><br>
<br>
</div>I suppose that's a start. If you're willing/able to create and run a<br>
modified kernel for the purpose of collecting more debug info, perhaps<br>
we can get a bit more creative in collecting more data on the problem<br>
(but a stack trace there is a good start).<br>
<br>
BTW- you might want to place the call after the XFS_FORCED_SHUTDOWN(mp)<br>
check almost halfway into the function to avoid duplicate messages.<br>
<br>
Brian<br>
<br>
><br>
> 2013/4/12 ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>>><br>
<div>><br>
> Hi Brian,<br>
> What else I'm missing? Thank you.<br>
> stap -e 'probe module("xfs").function("xfs_iunlink"){}'<br>
><br>
> WARNING: cannot find module xfs debuginfo: No DWARF information found<br>
> semantic error: no match while resolving probe point<br>
> module("xfs").function("xfs_iunlink")<br>
> Pass 2: analysis failed. Try again with another '--vp 01' option.<br>
><br>
><br>
</div>> 2013/4/12 ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a> <mailto:<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>>><br>
<div>><br>
> ls -l<br>
> /usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/kernel/fs/xfs/xfs.ko.debug<br>
> -r--r--r-- 1 root root 21393024 Apr 12 12:08<br>
> /usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/kernel/fs/xfs/xfs.ko.debug<br>
><br>
> rpm -qa|grep kernel<br>
> kernel-headers-2.6.32-279.el6.x86_64<br>
> kernel-devel-2.6.32-279.el6.x86_64<br>
> kernel-2.6.32-358.el6.x86_64<br>
> kernel-debuginfo-common-x86_64-2.6.32-279.el6.x86_64<br>
> abrt-addon-kerneloops-2.0.8-6.el6.x86_64<br>
> kernel-firmware-2.6.32-358.el6.noarch<br>
> kernel-debug-2.6.32-358.el6.x86_64<br>
> kernel-debuginfo-2.6.32-279.el6.x86_64<br>
> dracut-kernel-004-283.el6.noarch<br>
> libreport-plugin-kerneloops-2.0.9-5.el6.x86_64<br>
> kernel-devel-2.6.32-358.el6.x86_64<br>
> kernel-2.6.32-279.el6.x86_64<br>
><br>
> rpm -q kernel-debuginfo<br>
> kernel-debuginfo-2.6.32-279.el6.x86_64<br>
><br>
> rpm -q kernel<br>
> kernel-2.6.32-279.el6.x86_64<br>
> kernel-2.6.32-358.el6.x86_64<br>
><br>
> do I need to re probe it?<br>
><br>
><br>
> 2013/4/12 Eric Sandeen <<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a><br>
</div>> <mailto:<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a>>><br>
<div><div>><br>
> On 4/11/13 11:32 PM, ·ûÓÀÌÎ wrote:<br>
> > Hi Brian,<br>
> > Sorry but when I execute the script it says:<br>
> > WARNING: cannot find module xfs debuginfo: No DWARF<br>
> information found<br>
> > semantic error: no match while resolving probe point<br>
> module("xfs").function("xfs_iunlink")<br>
> ><br>
> > uname -a<br>
> > 2.6.32-279.el6.x86_64<br>
> > kernel debuginfo has been installed.<br>
> ><br>
> > Where can I find the correct xfs debuginfo?<br>
><br>
> it should be in the kernel-debuginfo rpm (of the same<br>
> version/release as the kernel rpm you're running)<br>
><br>
> You should have:<br>
><br>
> /usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/kernel/fs/xfs/xfs.ko.debug<br>
><br>
> If not, can you show:<br>
><br>
> # uname -a<br>
> # rpm -q kernel<br>
> # rpm -q kernel-debuginfo<br>
><br>
> -Eric<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> ·ûÓÀÌÎ<br>
><br>
><br>
><br>
><br>
> --<br>
> ·ûÓÀÌÎ<br>
><br>
><br>
><br>
><br>
> --<br>
> ·ûÓÀÌÎ<br>
><br>
><br>
</div></div><div><div>> _______________________________________________<br>
> xfs mailing list<br>
> <a href="mailto:xfs@oss.sgi.com" target="_blank">xfs@oss.sgi.com</a><br>
> <a href="http://oss.sgi.com/mailman/listinfo/xfs" target="_blank">http://oss.sgi.com/mailman/listinfo/xfs</a><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>