[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

xfs_freeze stuck on filesystem with several snapshots.



I'm looking at a machine where xfs_freeze -f is stuck in D state.  The
freeze is done on a filesystem that already has three LVM snapshots, and has
writes streamed to it over both nfs and samba.  I have some backtraces from
kdb (attached), but I'm not quite sure what to make of them.  After
xfs_freeze -f stuck, xfs_freeze -u was tried from the command line, and also
stuck.

The kernel is from the XFS CVS tree of Mar 19th, using LVM's VFS lock patch.
It has been patched to exempt certain processes coming from being stopped by
xfs_check_frozen, specifically kupdated and xfs_freeze.  knfsd and many smbd
processes are stopped by xfs_check_frozen, which is fine.  

kupdated seems to be stuck in a semaphore I don't recognize, descending out
of linvfs_pb_map()
kdb> btp 6
    EBP       EIP         Function(args)
0xefef5bbc 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0xc4ea6920,
0xcf5f221c, 0xefef5bd8)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xefef5be8 0xc0105cbd __down+0x61 (0xcf5f2214, 0xefef5e1c, 0x0)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xefef5bfc 0xc0105e2b __down_failed+0xb (0xcf5f21a0, 0xefef5c18, 0xc01c44b6,
0xcf5f21a0, 0xcf5f21a0)
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xefef5c08 0xc01d875b _text_lock_page_buf_locking+0x2d (0xc03d9420, 0x1,
0xd1c3f980, 0x8, 0x1)
                               kernel .text 0xc0100000 0xc01d872e 0xc01d8770
           0xc01f9b5b generic_make_request+0x97 (0x1, 0xd1c3f980,
0xd1c3f980, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01f9ac4 0xc01f9be0
[rest of trace is in attachment]

xfs_freeze -f is in pagebuf_iorequest().  How can I tell what it is waiting
for?

kdb> btp 2533
    EBP       EIP         Function(args)
0xc4d87b60 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0x0,
0xcf5f2248, 0x0)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xc4d87b98 0xc01d5dfe pagebuf_iorequest+0xae (0xcf5f21a0)
                               kernel .text 0xc0100000 0xc01d5d50 0xc01d5e88
0xc4d87ba4 0xc01defaa xfsbdstrat+0x2a (0xcfdebc00, 0xcf5f21a0, 0xcfc2c820,
0xcfdebc00, 0xc4d86000)
                               kernel .text 0xc0100000 0xc01def80 0xc01defbc
0xc4d87bc4 0xc01c3fb6 xfs_unmountfs_writesb+0xc2 (0xcfdebc00, 0xcfdebc00,
0xc4d86000, 0xcf5a6c80, 0xcfdebc00)
                               kernel .text 0xc0100000 0xc01c3ef4 0xc01c401c
0xc4d87be4 0xc01b0c44 xfs_fs_freeze+0xbc (0xcfdebc00, 0xbffffcac,
0xcf5a6c80, 0xcf5a6da4, 0x400151ac)
                               kernel .text 0xc0100000 0xc01b0b88 0xc01b0c60
[rest of trace is in attachment]

The attempt to jog things loose with xfs_freeze -u failed.  It seems to be
stuck on the same semaphore as kupdated.

kdb> btp 2814
    EBP       EIP         Function(args)
0xe1641b50 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0x0,
0xcf5f221c, 0xe1641b6c)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xe1641b7c 0xc0105cbd __down+0x61 (0xcf5f2214, 0x1000100, 0x0)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xe1641b90 0xc0105e2b __down_failed+0xb (0xcf5f21a0, 0xe1641bac, 0xc01c44b6,
0xcf5f21a0, 0x100)
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xe1641b9c 0xc01d875b _text_lock_page_buf_locking+0x2d (0xc476c560,
0x400f1000,
0x0, 0x400f1480, 0xed180be0)
                               kernel .text 0xc0100000 0xc01d872e 0xc01d8770
[rest of trace in attachment]

Any hints, suggestions or explanations would be very much appreciated.  I
still have access to the machine at the moment.

Thank you,
Dale Stephenson
steph@snapserver.com

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         6  0.0  0.0     0    0 ?        DW   May20   0:07 [kupdated]
root       204  0.0  0.0  1392  552 ?        S    May20   0:00 /sbin/LEDdrvd
root       218  0.0  0.0  1420  580 ?        S    May20   0:00 /sbin/LCDmgrd
root       888  0.0  0.0     0    0 ?        DW   May20   0:24 [nfsd]
root       974  0.0  0.2  5424 2140 ?        S    May20   0:00 smbd -D
kkint     1692  0.0  0.3  6084 3060 ?        D    May20   0:23 smbd -D
root      2533  0.0  0.0  1300  328 ?        D    May20   0:00 /usr/sbin/xfs_fre
kkint     2687  0.0  0.3  5956 2764 ?        D    May20   0:00 smbd -D
root      2814  0.0  0.0  1300  328 ttyp0    D    May20   0:00 xfs_freeze -u /hd
kkint     2823  0.0  0.3  5956 2704 ?        S    May20   0:00 smbd -D
kkint     2942  0.0  0.3  5956 2704 ?        S    May20   0:00 smbd -D
kkint     3061  0.0  0.3  5996 2764 ?        S    May20   0:00 smbd -D
kkint     3180  0.0  0.3  6004 2772 ?        D    May20   0:00 smbd -D
kkint     3308  0.0  0.3  6004 2760 ?        D    May20   0:00 smbd -D
kkint     3440  0.0  0.3  6004 2764 ?        D    May20   0:00 smbd -D
kkint     3559  0.0  0.3  5988 2740 ?        D    May20   0:00 smbd -D
kkint     3651  0.0  0.3  5988 2740 ?        D    May20   0:00 smbd -D
kkint    10701  0.0  0.3  6004 2816 ?        D    May20   0:00 smbd -D
kkint    11319  0.0  0.3  5980 2752 ?        D    May20   0:00 smbd -D
kkint    11416  0.0  0.3  6012 2824 ?        D    May20   0:00 smbd -D
kkint    11517  0.0  0.3  5988 2756 ?        D    May20   0:00 smbd -D
kkint    11614  0.0  0.3  6012 2820 ?        D    May20   0:00 smbd -D
kkint    11713  0.0  0.3  6004 2748 ?        D    May20   0:00 smbd -D
kkint    11804  0.0  0.3  6004 2788 ?        D    May20   0:00 smbd -D
kkint    11913  0.0  0.3  6004 2768 ?        D    May20   0:00 smbd -D
kkint    12014  0.0  0.3  6012 2792 ?        D    May20   0:00 smbd -D
kkint    12107  0.0  0.3  6012 2768 ?        D    May20   0:00 smbd -D
kkint    12229  0.0  0.3  6036 2836 ?        D    May20   0:00 smbd -D
kkint    12330  0.0  0.3  6012 2772 ?        D    May20   0:00 smbd -D
kkint    12431  0.0  0.3  6036 2840 ?        D    May20   0:00 smbd -D
kkint    12532  0.0  0.3  6020 2788 ?        D    May20   0:00 smbd -D
kkint    12638  0.0  0.3  6044 2844 ?        D    May20   0:00 smbd -D
kkint    12739  0.0  0.3  6036 2748 ?        D    May20   0:00 smbd -D
kkint    12840  0.0  0.3  6036 2812 ?        D    May20   0:00 smbd -D
kkint    12941  0.0  0.3  6036 2788 ?        D    May20   0:00 smbd -D
kkint    13042  0.0  0.3  6036 2816 ?        D    May20   0:00 smbd -D
kkint    13143  0.0  0.3  6036 2780 ?        D    May20   0:00 smbd -D
kkint    13257  0.0  0.3  6036 2776 ?        S    May20   0:00 smbd -D
kkint    13358  0.0  0.3  6044 2776 ?        S    May20   0:00 smbd -D
kkint    13456  0.0  0.3  6044 2772 ?        S    May20   0:00 smbd -D
kkint    13557  0.0  0.3  6044 2820 ?        D    May20   0:00 smbd -D
kkint    13658  0.0  0.3  6044 2788 ?        D    May20   0:00 smbd -D
kkint    13759  0.0  0.3  6060 2776 ?        D    May20   0:00 smbd -D
kkint    13860  0.0  0.3  6060 2796 ?        D    May20   0:00 smbd -D
kkint    13961  0.0  0.3  6060 2832 ?        D    May20   0:00 smbd -D
kkint    14070  0.0  0.3  6068 2808 ?        D    May20   0:00 smbd -D
kkint    14169  0.0  0.3  6068 2840 ?        D    May20   0:00 smbd -D
kkint    14283  0.0  0.3  6068 2804 ?        D    May20   0:00 smbd -D
kkint    14382  0.0  0.3  6068 2836 ?        D    May20   0:00 smbd -D
kkint    14493  0.0  0.3  6068 2808 ?        D    May20   0:00 smbd -D
root     22445  0.0  0.0  1432  528 ttyS0    R    07:37   0:00 grep D

kdb> btp 6
    EBP       EIP         Function(args)
0xefef5bbc 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0xc4ea6920, 0xcf5f221c, 0xefef5bd8)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xefef5be8 0xc0105cbd __down+0x61 (0xcf5f2214, 0xefef5e1c, 0x0)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xefef5bfc 0xc0105e2b __down_failed+0xb (0xcf5f21a0, 0xefef5c18, 0xc01c44b6, 0xcf5f21a0, 0xcf5f21a0)
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xefef5c08 0xc01d875b _text_lock_page_buf_locking+0x2d (0xc03d9420, 0x1, 0xd1c3f980, 0x8, 0x1)
                               kernel .text 0xc0100000 0xc01d872e 0xc01d8770
           0xc01f9b5b generic_make_request+0x97 (0x1, 0xd1c3f980, 0xd1c3f980, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01f9ac4 0xc01f9be0
0xefef5e20 0xc01f9c2f submit_bh+0x4f (0xd231124c, 0x3f000, 0x0, 0x1000, 0x10002)                               kernel .text 0xc0100000 0xc01f9be0 0xc01f9c50
0xefef5e60 0xc01dbcaf linvfs_pb_bmap+0x77 (0xd2cb1d60, 0x3f000, 0x0, 0x1000, 0xefef5e98)
                               kernel .text 0xc0100000 0xc01dbc38 0xc01dbd04
0xefef5eb8 0xc01d819c pagebuf_delalloc_convert+0x5c (0xd2cb1d60, 0xc14b6b80, 0x10002, 0xc01dbc38, 0x1)
                               kernel .text 0xc0100000 0xc01d8140 0xc01d8200
0xefef5efc 0xc01d7444 pagebuf_write_full_page+0xa0 (0xc14b6b80, 0xc01dbc38, 0xd215d620, 0xc14b6b80)
                               kernel .text 0xc0100000 0xc01d73a4 0xc01d7488
0xefef5f14 0xc01dbe85 linvfs_write_full_page+0x45 (0xc14b6b80, 0xd215d620, 0x0)
                               kernel .text 0xc0100000 0xc01dbe40 0xc01dbeac
0xefef5f28 0xc012ec66 write_buffer_delay+0x56 (0xd215d620, 0xefef4000, 0xefef4570, 0xefef4560, 0xc7b9)
more>
                               kernel .text 0xc0100000 0xc012ec10 0xc012ec70
0xefef5fc8 0xc012ee71 write_some_buffers+0x75 (0x0, 0xefef4000, 0xc034a000, 0xefef4560, 0xefef4000)
                               kernel .text 0xc0100000 0xc012edfc 0xc012eef4
           0xc01082f3 do_IRQ+0x87
                               kernel .text 0xc0100000 0xc010826c 0xc0108314
           0xc010577b kernel_thread+0x23
                               kernel .text 0xc0100000 0xc0105758 0xc0105790

kdb> btp 888
    EBP       EIP         Function(args)
0xed1bdcd4 0xc0110d5f schedule+0x33f (0x202, 0xcfdebc00, 0x1, 0x1000, 0xed1bdd00)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xed1bdd10 0xc01e2448 _sv_wait+0xbc (0xcfdebf78, 0xcfdebf70, 0x202, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01e238c 0xc01e2464
0xed1bdd3c 0xc01c4941 xfs_check_frozen+0xa5 (0xcfdebc00, 0xc9a78c44, 0x8001, 0x1, 0x8f2000)
                               kernel .text 0xc0100000 0xc01c489c 0xc01c4980
0xed1bdda8 0xc01dca83 xfs_write+0x87 (0xc9a78c44, 0xed1bde00, 0x8001, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01dc9fc 0xc01dcea4
0xed1bde20 0xc01d8d0c linvfs_write+0x2ec (0xed1bde80, 0xc35940e0, 0x2000, 0xed1bdea0, 0x0)
                               kernel .text 0xc0100000 0xc01d8a20 0xc01d8d48
0xed1bdee8 0xc01677af nfsd_write+0xf7 (0xed240a60, 0xed1bdeec, 0x88, 0xed1bc000, 0x0)
                               kernel .text 0xc0100000 0xc01676b8 0xc0167920
0xed1bde80 0xc0259074 inet_sendmsg+0x40 (0xcf6e4000, 0xcf6e4200, 0xcf6e4400, 0xcf6e4138, 0xc033c2c0)
                               kernel .text 0xc0100000 0xc0259034 0xc025907c
0xed1bdf3c 0xc0163eab nfsd_dispatch+0xc7 (0xcf6e4000, 0xcf9d8014, 0xf1db, 0xed1bc000, 0xcf0368a0)
                               kernel .text 0xc0100000 0xc0163de4 0xc0163f70
0xed1bdfb0 0xc026f56a svc_process+0x2b6 (0xcf6e4000, 0xfffffff5, 0xed1bc000, 0xcf0368a0, 0xc033c180)
                               kernel .text 0xc0100000 0xc026f2b4 0xc026f7d0
0xed1bdfac 0xc0270b7d svc_recv+0x27d
                               kernel .text 0xc0100000 0xc0270900 0xc0270cf8
           0xc010577b kernel_thread+0x23
more>
                               kernel .text 0xc0100000 0xc0105758 0xc0105790

kdb> btp 1692
    EBP       EIP         Function(args)
0xca9dbe40 0xc0110d5f schedule+0x33f (0x202, 0xcfdebc00, 0x1, 0xcbf4cbd8, 0xca9dbe6c)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xca9dbe7c 0xc01e2448 _sv_wait+0xbc (0xcfdebf78, 0xcfdebf70, 0x202, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01e238c 0xc01e2464
0xca9dbea8 0xc01c4941 xfs_check_frozen+0xa5 (0xcfdebc00, 0xe6d8124c, 0x28001, 0x1, 0x2d000)
                               kernel .text 0xc0100000 0xc01c489c 0xc01c4980
0xca9dbf14 0xc01dca83 xfs_write+0x87 (0xe6d8124c, 0xca9dbf6c, 0x28001, 0x0, 0x0)                               kernel .text 0xc0100000 0xc01dc9fc 0xc01dcea4
0xca9dbf8c 0xc01d8d0c linvfs_write+0x2ec (0xecbffc20, 0x402c904c, 0xf000, 0xecbffc40, 0xca9da000)
                               kernel .text 0xc0100000 0xc01d8a20 0xc01d8d48
0xca9dbfbc 0xc012df88 sys_write+0xa4 (0x1d, 0x402c904c, 0xf000, 0x82f15a0, 0xf000)
                               kernel .text 0xc0100000 0xc012dee4 0xc012dfe4
           0xc0106ecb system_call+0x33
                               kernel .text 0xc0100000 0xc0106e98 0xc0106ed0

kdb> btp 2533
    EBP       EIP         Function(args)
0xc4d87b60 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0x0, 0xcf5f2248, 0x0)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xc4d87b98 0xc01d5dfe pagebuf_iorequest+0xae (0xcf5f21a0)
                               kernel .text 0xc0100000 0xc01d5d50 0xc01d5e88
0xc4d87ba4 0xc01defaa xfsbdstrat+0x2a (0xcfdebc00, 0xcf5f21a0, 0xcfc2c820, 0xcfdebc00, 0xc4d86000)
                               kernel .text 0xc0100000 0xc01def80 0xc01defbc
0xc4d87bc4 0xc01c3fb6 xfs_unmountfs_writesb+0xc2 (0xcfdebc00, 0xcfdebc00, 0xc4d86000, 0xcf5a6c80, 0xcfdebc00)
                               kernel .text 0xc0100000 0xc01c3ef4 0xc01c401c
0xc4d87be4 0xc01b0c44 xfs_fs_freeze+0xbc (0xcfdebc00, 0xbffffcac, 0xcf5a6c80, 0xcf5a6da4, 0x400151ac)
                               kernel .text 0xc0100000 0xc01b0b88 0xc01b0c60
0xc4d87f44 0xc01db064 xfs_ioctl+0x13d4 (0xcbf4cb00, 0x0, 0xcf14ce60, 0xee02210a, 0xcbf4cb00)
                               kernel .text 0xc0100000 0xc01d9c90 0xc01db150
0xc4d87d70 0xc02475c7 tcp_rcv_established+0x36b (0xcf14ce60, 0xcfedc800, 0xc86ff030, 0xcf14ce60, 0x6)
                               kernel .text 0xc0100000 0xc024725c 0xc02479bc
0xc4d87df8 0xc02389b3 ip_local_deliver+0xdb (0xcf14ce60, 0xcf14ce60, 0x0, 0xcf14ce60, 0xcf2803a0)
                               kernel .text 0xc0100000 0xc02388d8 0xc0238a28
0xc4d87e14 0xc0238d1e ip_rcv+0x2f6 (0xcf14ce60, 0xcfedc800, 0x282, 0xc03394bc, 0x1ff)
                               kernel .text 0xc0100000 0xc0238a28 0xc0238d80
0xc4d87e44 0xc022da2d net_rx_action+0x129 (0xecd4fd80, 0x400f1000, 0x0, 0x400f1480, 0xef161ee0)
                               kernel .text 0xc0100000 0xc022d904 0xc022db04
0xc4d87ec4 0xc011fc58 do_no_page+0x48 (0xef161ee0, 0xecd4fd80, 0x400f1480, 0x0,
0xc40ff3c4)
more>
                               kernel .text 0xc0100000 0xc011fc10 0xc011fd2c
0xc4d87ef0 0xc011fd82 handle_mm_fault+0x56 (0xef161ee0, 0xecd4fd80, 0x400f1480,
0x0, 0xc4d86000)
                               kernel .text 0xc0100000 0xc011fd2c 0xc011fde0
0xc4d87fb4 0xc01101cb do_page_fault+0x1af (0x30002, 0xc4d87f7c)
                               kernel .text 0xc0100000 0xc011001c 0xc01104d6
0xc4d87f54 0xc012c5f0 vfs_statfs+0x40 (0xcf6bd20c, 0xcf5a6c80, 0xe0e73c60, 0xc0045877, 0xbffffcac)
                               kernel .text 0xc0100000 0xc012c5b0 0xc012c600
0xc4d87f70 0xc01d90cd linvfs_ioctl+0x2d (0xcf5a6c80, 0xe0e73c60, 0xc0045877, 0xbffffcac, 0xc4d86000)
                               kernel .text 0xc0100000 0xc01d90a0 0xc01d90f2
0xc4d87fbc 0xc013ab0a sys_ioctl+0x26a (0xbffffd28, 0xc0106ecb, 0x9, 0xc0045877,
0xbffffcac)
                               kernel .text 0xc0100000 0xc013a8a0 0xc013ab30
           0xc0106fbc error_code+0x34
                               kernel .text 0xc0100000 0xc0106f88 0xc0106fc4
Interrupt registers:
eax = 0x00000001 ebx = 0xbffffd28 ecx = 0xc0106ecb edx = 0x00000009
esi = 0xc0045877 edi = 0xbffffcac esp = 0x00000023 eip = 0x0000002b
ebp = 0xbffffd94 xss = 0x00000246 xcs = 0x00000036 eflags = 0x400f1494
xds = 0xbffffd28 xes = 0x00000036 origeax = 0x400e002b &regs = 0xc4d87fbc
Interrupt from user space, end of kernel trace

kdb> btp 2687
    EBP       EIP         Function(args)
0xe3f03f10 0xc0110d5f schedule+0x33f (0xe4d45920, 0xe4d4598c, 0xcff2a420, 0xe4d45994, 0xe3f03f2c)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xe3f03f3c 0xc0105cbd __down+0x61 (0xe4d4598c, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xe3f03f50 0xc0105e2b __down_failed+0xb
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xe3f03f8c 0xc012db0d _text_lock_open+0x5
                               kernel .text 0xc0100000 0xc012db08 0xc012db20

kdb> btp 2814
    EBP       EIP         Function(args)
0xe1641b50 0xc0110d5f schedule+0x33f (0xcf5f21a0, 0xcfdebc00, 0x0, 0xcf5f221c, 0xe1641b6c)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xe1641b7c 0xc0105cbd __down+0x61 (0xcf5f2214, 0x1000100, 0x0)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xe1641b90 0xc0105e2b __down_failed+0xb (0xcf5f21a0, 0xe1641bac, 0xc01c44b6, 0xcf5f21a0, 0x100)
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xe1641b9c 0xc01d875b _text_lock_page_buf_locking+0x2d (0xc476c560, 0x400f1000,
0x0, 0x400f1480, 0xed180be0)
                               kernel .text 0xc0100000 0xc01d872e 0xc01d8770
           0xc011fc58 do_no_page+0x48 (0xed180be0, 0xc476c560, 0x400f1480, 0x0,
0xc4bd13c4)
                               kernel .text 0xc0100000 0xc011fc10 0xc011fd2c
0xe1641ef0 0xc011fd82 handle_mm_fault+0x56 (0xed180be0, 0xc476c560, 0x400f1480,
0x0, 0xe1640000)
                               kernel .text 0xc0100000 0xc011fd2c 0xc011fde0
0xe1641fb4 0xc01101cb do_page_fault+0x1af (0x30002, 0xe1641f7c)
                               kernel .text 0xc0100000 0xc011001c 0xc01104d6
0xe1641f54 0xc012c5f0 vfs_statfs+0x40 (0xcf6bd20c, 0xcf5a6c80, 0xe0e738e0, 0xc0045878, 0xbffffc5c)
                               kernel .text 0xc0100000 0xc012c5b0 0xc012c600
0xe1641f70 0xc01d90cd linvfs_ioctl+0x2d (0xcf5a6c80, 0xe0e738e0, 0xc0045878, 0xbffffc5c, 0xe1640000)
                               kernel .text 0xc0100000 0xc01d90a0 0xc01d90f2
0xe1641fbc 0xc013ab0a sys_ioctl+0x26a (0xbffffcd8, 0xc0106ecb, 0x3, 0xc0045878,
0xbffffc5c)
                               kernel .text 0xc0100000 0xc013a8a0 0xc013ab30
           0xc0106fbc error_code+0x34
more>
                               kernel .text 0xc0100000 0xc0106f88 0xc0106fc4
Interrupt registers:
eax = 0x00000000 ebx = 0xbffffcd8 ecx = 0xc0106ecb edx = 0x00000003
esi = 0xc0045878 edi = 0xbffffc5c esp = 0x00000023 eip = 0x0000002b
ebp = 0xbffffd44 xss = 0x00000246 xcs = 0x00000036 eflags = 0x400f1494
xds = 0xbffffcd8 xes = 0x00000036 origeax = 0x400e002b &regs = 0xe1641fbc
Interrupt from user space, end of kernel trace

kdb> btp 3180
    EBP       EIP         Function(args)
0xe4d25c24 0xc0110d5f schedule+0x33f (0x202, 0xcfdebc00, 0x2, 0x1, 0xe4d25c50)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xe4d25c60 0xc01e2448 _sv_wait+0xbc (0xcfdebf78, 0xcfdebf70, 0x202, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01e238c 0xc01e2464
0xe4d25c8c 0xc01c4941 xfs_check_frozen+0xa5 (0xcfdebc00, 0x0, 0x0, 0x2, 0x0)
                               kernel .text 0xc0100000 0xc01c489c 0xc01c4980
0xe4d25cac 0xc01c7e83 xfs_trans_alloc+0x17 (0xcfdebc00, 0xa, 0x1ff, 0xcc32a444,
0xcfe8513c)
                               kernel .text 0xc0100000 0xc01c7e6c 0xc01c7e94
0xe4d25d48 0xc01d0819 xfs_mkdir+0x14d (0xe2233058, 0xcfe8513c, 0xe4d25ec4, 0xe4d25d90, 0x0)
                               kernel .text 0xc0100000 0xc01d06cc 0xc01d0d14
0xe4d25f34 0xc01db2f9 linvfs_common_cr+0x15d (0xe2233058, 0xe4d25f54, 0xc01d902e, 0xc40f0000, 0xcc32a320)
                               kernel .text 0xc0100000 0xc01db19c 0xc01db404
           0xc01d8f39 linvfs_readdir+0x119 (0xcdcafa20, 0x809a000, 0x8, 0xe2233040, 0xe4d25ed0)
                               kernel .text 0xc0100000 0xc01d8e20 0xc01d9048
0xe4d25ec4 0xc011fc58 do_no_page+0x48 (0xcc32a320, 0xcfe850e0, 0x1ff, 0xcfe850e0, 0xcfe850e0)
                               kernel .text 0xc0100000 0xc011fc10 0xc011fd2c
0xe4d25f78 0xc01380da vfs_mkdir+0xc6 (0xcc32a320, 0xcfe850e0, 0x1ff, 0xe4d24000, 0x1ff)
                               kernel .text 0xc0100000 0xc0138014 0xc0138110
0xe4d25fbc 0xc01381ac sys_mkdir+0x9c (0xbffff410, 0x1ff, 0x0, 0x1ff, 0x82d7d20)
                               kernel .text 0xc0100000 0xc0138110 0xc01381f0
           0xc0106ecb system_call+0x33
more>
                               kernel .text 0xc0100000 0xc0106e98 0xc0106ed0
kdb> btp 3651 (3559, 3440, 3308)
    EBP       EIP         Function(args)
0xc7d41f20 0xc0110d5f schedule+0x33f (0xcc32a320, 0xfffffff7, 0xcc32a38c, 0xcc32a394, 0xc7d41f3c)
                               kernel .text 0xc0100000 0xc0110a20 0xc0110d88
0xc7d41f4c 0xc0105cbd __down+0x61 (0xcc32a38c, 0xc033d700, 0xffffffec)
                               kernel .text 0xc0100000 0xc0105c5c 0xc0105d08
0xc7d41f60 0xc0105e2b __down_failed+0xb
                               kernel .text 0xc0100000 0xc0105e20 0xc0105e34
0xc7d41f74 0xc013b14e _text_lock_readdir+0x5
                               kernel .text 0xc0100000 0xc013b149 0xc013b180