<div dir="ltr">The servers are back to service now and It's hard to run xfs_repair. It always happen bellow is the xfs_repair log when it happens on another server several days ago. <br> -sh-4.1$ sudo xfs_repair -n /dev/glustervg/glusterlv<br>
Phase 1 - find and verify superblock…<br>Phase 2 - using internal log<br> - scan filesystem freespace and inode maps…<br>agi unlinked bucket 0 is 4046848 in ag 0 (inode=4046848)<br>agi unlinked bucket 5 is 2340485 in ag 0 (inode=2340485)<br>
agi unlinked bucket 6 is 2326854 in ag 0 (inode=2326854)<br>agi unlinked bucket 8 is 1802120 in ag 0 (inode=1802120)<br>agi unlinked bucket 14 is 495566 in ag 0 (inode=495566)<br>agi unlinked bucket 16 is 5899536 in ag 0 (inode=5899536)<br>
agi unlinked bucket 19 is 4008211 in ag 0 (inode=4008211)<br>agi unlinked bucket 21 is 4906965 in ag 0 (inode=4906965)<br>agi unlinked bucket 23 is 2022231 in ag 0 (inode=2022231)<br>agi unlinked bucket 24 is 1626200 in ag 0 (inode=1626200)<br>
agi unlinked bucket 25 is 938585 in ag 0 (inode=938585)<br>agi unlinked bucket 30 is 4226526 in ag 0 (inode=4226526)<br>agi unlinked bucket 34 is 4108962 in ag 0 (inode=4108962)<br>agi unlinked bucket 37 is 1740389 in ag 0 (inode=1740389)<br>
agi unlinked bucket 39 is 247399 in ag 0 (inode=247399)<br>agi unlinked bucket 40 is 6237864 in ag 0 (inode=6237864)<br>agi unlinked bucket 43 is 3404331 in ag 0 (inode=3404331)<br>agi unlinked bucket 45 is 2092717 in ag 0 (inode=2092717)<br>
agi unlinked bucket 48 is 4041008 in ag 0 (inode=4041008)<br>agi unlinked bucket 50 is 1459762 in ag 0 (inode=1459762)<br>agi unlinked bucket 56 is 852024 in ag 0 (inode=852024)<br> - found root in ode chunk<br>Phase 3 - for each AG…<br>
- scan (but don't clear) agi unlinked lists…<br> - process known inodes and perform inode discovery…<br> - agno = 0<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 123696, len 16384 bytes) key=(bno 123696, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 247776, len 16384 bytes) key=(bno 247776, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 425984, len 16384 bytes) key=(bno 425984, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 469280, len 16384 bytes) key=(bno 469280, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 729856, len 16384 bytes) key=(bno 729856, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 813072, len 16384 bytes) key=(bno 813072, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 870176, len 16384 bytes) key=(bno 870176, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 901056, len 16384 bytes) key=(bno 901056, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 1011104, len 16384 bytes) key=(bno 1011104, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 1046336, len 16384 bytes) key=(bno 1046336, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 1163424, len 16384 bytes) key=(bno 1163424, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 1170240, len 16384 bytes) key=(bno 1170240, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 1702160, len 16384 bytes) key=(bno 1702160, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2004096, len 16384 bytes) key=(bno 2004096, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2020496, len 16384 bytes) key=(bno 2020496, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2023408, len 16384 bytes) key=(bno 2023408, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2054464, len 16384 bytes) key=(bno 2054464, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2113232, len 16384 bytes) key=(bno 2113232, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2453472, len 16384 bytes) key=(bno 2453472, len 8192 bytes)<br>
7f084d34e700: Badness in key lookup (length)<br>bp=(bno 2949760, len 16384 bytes) key=(bno 2949760, len 8192 bytes)<br>7f084d34e700: Badness in key lookup (length)<br>bp=(bno 3118912, len 16384 bytes) key=(bno 3118912, len 8192 bytes)<br>
- agno = 1<br> - agno = 2<br> - agno = 3<br> - agno = 4<br> - agno = 5<br> - agno = 6<br> - agno = 7<br> - agno = 8<br> - agno = 9<br> - agno = 10<br>
- agno = 11<br> - agno = 12<br> - agno = 13<br> - agno = 14<br> - agno = 15<br> - agno = 16<br> - agno = 17<br> - agno = 18<br> - agno = 19<br> - agno = 20<br>
- agno = 21<br> - agno = 22<br> - agno = 23<br> - agno = 24<br> - agno = 25<br> - agno = 26<br> - agno = 27<br> - agno = 28<br> - agno = 29<br> - agno = 30<br>
- process newly discovered in odes..<br>Phase 4 - check for duplicate blocks…<br> - setting up duplicate extent list…<br> - check for inodes claiming duplicate blocks…<br> - agno = 0<br> - agno = 1<br>
- agno = 3<br> - agno = 9<br> - agno = 12<br> - agno = 14<br> - agno = 5<br> - agno = 19<br> - agno = 23<br> - agno = 24<br> - agno = 25<br> - agno = 26<br>
- agno = 27<br> - agno = 28<br> - agno = 29<br> - agno = 30<br> - agno = 4<br> - agno = 2<br> - agno = 17<br> - agno = 6<br> - agno = 8<br> - agno = 16<br>
- agno = 11<br> - agno = 10<br> - agno = 18<br> - agno = 13<br> - agno = 15<br> - agno = 20<br> - agno = 22<br> - agno = 21<br> - agno = 7<br>No modify flag set, skipping phase 5<br>
Phase 6 - check inode connectivity…<br> - traversing filesystem …<br> - traversal finished …<br> - moving disconnected inodes to lost+found …<br>disconnected inode 6235944, would move to lost+found<br>
Phase 7 - verify link counts…<br>would have reset inode 6235944 nlinks from 0 to 1<br>No modify flag set, skipping filesystem flush and exiting..<br><br><br><br>µÚ¶þ²½<br>repairµÄlog<br><br>sh-4.1$ sudo xfs_repair /dev/glustervg/glusterlv<br>
Phase 1 - find and verify superblock…<br>Phase 2 - using internal log<br> - zero log…<br> - scan filesystem freespace and inode maps…<br>agi unlinked bucket 0 is 4046848 in ag 0 (inode=4046848)<br>agi unlinked bucket 5 is 2340485 in ag 0 (inode=2340485)<br>
agi unlinked bucket 6 is 2326854 in ag 0 (inode=2326854)<br>agi unlinked bucket 8 is 1802120 in ag 0 (inode=1802120)<br>agi unlinked bucket 14 is 495566 in ag 0 (inode=495566)<br>agi unlinked bucket 16 is 5899536 in ag 0 (inode=5899536)<br>
agi unlinked bucket 19 is 4008211 in ag 0 (inode=4008211)<br>agi unlinked bucket 21 is 4906965 in ag 0 (inode=4906965)<br>agi unlinked bucket 23 is 2022231 in ag 0 (inode=2022231)<br>agi unlinked bucket 24 is 1626200 in ag 0 (inode=1626200)<br>
agi unlinked bucket 25 is 938585 in ag 0 (inode=938585)<br>agi unlinked bucket 30 is 4226526 in ag 0 (inode=4226526)<br>agi unlinked bucket 34 is 4108962 in ag 0 (inode=4108962)<br>agi unlinked bucket 37 is 1740389 in ag 0 (inode=1740389)<br>
agi unlinked bucket 39 is 247399 in ag 0 (inode=247399)<br>agi unlinked bucket 40 is 6237864 in ag 0 (inode=6237864)<br>agi unlinked bucket 43 is 3404331 in ag 0 (inode=3404331)<br>agi unlinked bucket 45 is 2092717 in ag 0 (inode=2092717)<br>
agi unlinked bucket 48 is 4041008 in ag 0 (inode=4041008)<br>agi unlinked bucket 50 is 1459762 in ag 0 (inode=1459762)<br>agi unlinked bucket 56 is 852024 in ag 0 (inode=852024)<br> - found root in ode chunk<br>Phase 3 - for each AG…<br>
- scan and clear agi unlinked lists…<br> - process known inodes and perform inode discovery…<br> - agno = 0<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 123696, len 16384 bytes) key=(bno 123696, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 247776, len 16384 bytes) key=(bno 247776, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 425984, len 16384 bytes) key=(bno 425984, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 469280, len 16384 bytes) key=(bno 469280, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 729856, len 16384 bytes) key=(bno 729856, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 813072, len 16384 bytes) key=(bno 813072, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 870176, len 16384 bytes) key=(bno 870176, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 901056, len 16384 bytes) key=(bno 901056, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 1011104, len 16384 bytes) key=(bno 1011104, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 1046336, len 16384 bytes) key=(bno 1046336, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 1163424, len 16384 bytes) key=(bno 1163424, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 1170240, len 16384 bytes) key=(bno 1170240, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 1702160, len 16384 bytes) key=(bno 1702160, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2004096, len 16384 bytes) key=(bno 2004096, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2020496, len 16384 bytes) key=(bno 2020496, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2023408, len 16384 bytes) key=(bno 2023408, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2054464, len 16384 bytes) key=(bno 2054464, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2113232, len 16384 bytes) key=(bno 2113232, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2453472, len 16384 bytes) key=(bno 2453472, len 8192 bytes)<br>
7f8220be6700: Badness in key lookup (length)<br>bp=(bno 2949760, len 16384 bytes) key=(bno 2949760, len 8192 bytes)<br>7f8220be6700: Badness in key lookup (length)<br>bp=(bno 3118912, len 16384 bytes) key=(bno 3118912, len 8192 bytes)<br>
- agno = 1<br> - agno = 2<br> - agno = 3<br> - agno = 4<br> - agno = 5<br> - agno = 6<br> - agno = 7<br> - agno = 8<br> - agno = 9<br> - agno = 10<br>
- agno = 11<br> - agno = 12<br> - agno = 13<br> - agno = 14<br> - agno = 15<br> - agno = 16<br> - agno = 17<br> - agno = 18<br> - agno = 19<br> - agno = 20<br>
- agno = 21<br> - agno = 22<br> - agno = 23<br> - agno = 24<br> - agno = 25<br> - agno = 26<br> - agno = 27<br> - agno = 28<br> - agno = 29<br> - agno = 30<br>
- process newly discovered in odes..<br>Phase 4 - check for duplicate blocks…<br> - setting up duplicate extent list…<br> - check for inodes claiming duplicate blocks…<br> - agno = 0<br> - agno = 4<br>
- agno = 2<br> - agno = 3<br> - agno = 7<br> - agno = 18<br> - agno = 28<br> - agno = 6<br> - agno = 5<br> - agno = 1<br> - agno = 26<br> - agno = 8<br>
- agno = 14<br> - agno = 17<br> - agno = 16<br> - agno = 10<br> - agno = 20<br> - agno = 13<br> - agno = 15<br> - agno = 11<br> - agno = 19<br> - agno = 22<br>
- agno = 21<br> - agno = 23<br> - agno = 9<br> - agno = 12<br> - agno = 24<br> - agno = 27<br> - agno = 25<br> - agno = 29<br> - agno = 30<br>Phase 5 - rebuild AG headers and trees…<br>
- reset superblock…<br>Phase 6 - check inode connectivity…<br> - resetting contents of realtime bitmap and summary in odes<br> - traversing filesystem …<br> - traversal finished …<br> - moving disconnected inodes to lost+found …<br>
disconnected inode 6235944, moving to lost+found<br>Phase 7 - verify and correct link counts…<br>done<br>sh-4.1$ .<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/9 Eric Sandeen <span dir="ltr"><<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 4/9/13 7:53 AM, ·ûÓÀÌÎ wrote:<br>
> Dear xfs experts,<br>
> I really need your help sincerely!!! In our production enviroment we<br>
> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file<br>
> system crash on some of the server frequently about every two weeks.<br>
> Can you help to give me a direction about how to debug this issue and<br>
> how to avoid it? Thank you very very much!<br>
<br>
</div>So this happens reliably, but infrequently? (only every 2 weeks or so?)<br>
<br>
Can you provoke it any more often?<br>
<div class="im"><br>
> uname -a<br>
> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64<br>
> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux<br>
<br>
</div>That's a RHEL6 kernel; I'm assuming that this is a RHEL clone w/o RH support?<br>
<br>
I agree with Ben that I'd like to see xfs_repair output.<br>
<br>
Since the fs has shut down, you should unmount, remount, and unmount<br>
again to replay the dirty log. Then do xfs_repair -n, and provide the output<br>
if it discovers any errors.<br>
<br>
Thanks,<br>
-Eric<br>
<div class="HOEnZb"><div class="h5"><br>
> Every time the crash log is same, as following<br>
><br>
> 038 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:<br>
> xfs_inotobp() returned error 22.<br>
> 1039 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree<br>
> returned error 22<br>
> 1040 Apr 9 09:41:36 cqdx kernel: XFS (sdb):<br>
> xfs_do_force_shutdown(0x1) called from line 1184 of file<br>
> fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02ee20a<br>
> 1041 Apr 9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.<br>
> Shutting down filesystem<br>
> 1042 Apr 9 09:41:36 cqdx kernel: XFS (sdb): Please umount the<br>
> filesystem and rectify the problem(s)<br>
> 1043 Apr 9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
> 1044 Apr 9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
> 1045 Apr 9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
> 1046 Apr 9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>