-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
hello,
i'm having a strange problem with our new storage cluster, i've read
nearly every xfs related threads (which contained:"called from line 1164
of file fs/xfs/xfs_trans.c" - i've surprised that the line number gives
very accurate results - and helps to omit old problems )
i've rsynced more than 4T of data into the system (and i've hit the
xfs-rsync bug, but it's working now without any problems)
problem occurs when copying simultaniously from windows/linux boxes to
the filesystem thru samba causes the crashes.
with older kernels the whole system crashed, with circular locking
problems (similar:http://oss.sgi.com/archives/xfs/2008-08/msg00354.html
) but with 2.6.27-rc4, it's just shutdows the filesystem, and i'm able
to remount it.
the biggest problem is that i can't cause the system to crash with tests
- i'm currently copying kernel trees in parrallel
i'm not sure that this is an xfs bug, because: rsync worked, and when i
tweaked the proc values and run test after test, it doesn't crashed.
the setup is:
node:(x8)
kernel:2.6.27-rc4
raid5
dmcrypt
iscsi_target (0.4.16)
master:
kernel:2.6.27-rc4
openiscsid (2.0-870)
raid5
xfs
samba (3.0.24-6etch10)
$ xfs_info /dev/md3
meta-data=/dev/md3 isize=256 agcount=128, agsize=26718592 blks
= sectsz=4096 attr=1
data = bsize=4096 blocks=3418704352, imaxpct=25
= sunit=128 swidth=896 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=2
= sectsz=4096 sunit=1 blks
realtime =none extsz=458752 blocks=0, rtextents=0
- --
Nagy Zoltan (kirk) <kirk@xxxxxxxx>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkix2fsACgkQLcjF8xTqkoBrvwCg22IjkGT3WRVNCRBIDp56CTNw
uZYAoK7pImMY7efqaxwKqhV0H5hDYdUT
=Zg5Z
-----END PGP SIGNATURE-----
XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1736 of file
fs/xfs/xfs_bmap.c. Caller 0xc034b05f
Pid: 18801, comm: pdflush Not tainted 2.6.27-rc4 #4
[<c03473f8>] xfs_bmap_add_extent_unwritten_real+0x1498/0x16a0
[<c034b05f>] xfs_bmap_add_extent+0x45f/0x560
[<c033a127>] xfs_alloc_vextent+0x267/0x4f0
[<c038704c>] xfs_trans_log_inode+0x1c/0x50
[<c034b05f>] xfs_bmap_add_extent+0x45f/0x560
[<c034f46b>] xfs_bmapi+0x9db/0x15f0
[<c0344cd8>] xfs_bmap_search_multi_extents+0x98/0xe0
[<c0372fae>] xfs_iomap_write_allocate+0x2de/0x490
[<c0374114>] xfs_iomap+0x334/0x410
[<c038f784>] xfs_map_blocks+0x44/0x90
[<c0390c16>] xfs_page_state_convert+0x536/0x790
[<c0390fb0>] xfs_vm_writepage+0x60/0x100
[<c01610a8>] __writepage+0x8/0x30
[<c0161615>] write_cache_pages+0x225/0x340
[<c01610a0>] __writepage+0x0/0x30
[<c03f8cb3>] submit_bio+0x63/0xf0
[<c0161750>] generic_writepages+0x20/0x30
[<c016178b>] do_writepages+0x2b/0x50
[<c01a1196>] __writeback_single_inode+0x86/0x310
[<c0385dc6>] xfs_trans_first_ail+0x16/0x30
[<c0376bca>] xfs_log_need_covered+0x6a/0xb0
[<c01a17fe>] generic_sync_sb_inodes+0x1de/0x2c0
[<c01a1c77>] writeback_inodes+0x87/0xb0
[<c0162135>] wb_kupdate+0x85/0xf0
[<c0162590>] pdflush+0x0/0x1b0
[<c016267e>] pdflush+0xee/0x1b0
[<c01620b0>] wb_kupdate+0x0/0xf0
[<c01387c2>] kthread+0x42/0x70
[<c0138780>] kthread+0x0/0x70
[<c0103c1b>] kernel_thread_helper+0x7/0x1c
=======================
Filesystem "md3": XFS internal error xfs_trans_cancel at line 1164 of file
fs/xfs/xfs_trans.c. Caller 0xc0373074
Pid: 18801, comm: pdflush Not tainted 2.6.27-rc4 #4
[<c0385439>] xfs_trans_cancel+0xe9/0x110
[<c0373074>] xfs_iomap_write_allocate+0x3a4/0x490
[<c0373074>] xfs_iomap_write_allocate+0x3a4/0x490
[<c0374114>] xfs_iomap+0x334/0x410
[<c038f784>] xfs_map_blocks+0x44/0x90
[<c0390c16>] xfs_page_state_convert+0x536/0x790
[<c0390fb0>] xfs_vm_writepage+0x60/0x100
[<c01610a8>] __writepage+0x8/0x30
[<c0161615>] write_cache_pages+0x225/0x340
[<c01610a0>] __writepage+0x0/0x30
[<c03f8cb3>] submit_bio+0x63/0xf0
[<c0161750>] generic_writepages+0x20/0x30
[<c016178b>] do_writepages+0x2b/0x50
[<c01a1196>] __writeback_single_inode+0x86/0x310
[<c0385dc6>] xfs_trans_first_ail+0x16/0x30
[<c0376bca>] xfs_log_need_covered+0x6a/0xb0
[<c01a17fe>] generic_sync_sb_inodes+0x1de/0x2c0
[<c01a1c77>] writeback_inodes+0x87/0xb0
[<c0162135>] wb_kupdate+0x85/0xf0
[<c0162590>] pdflush+0x0/0x1b0
[<c016267e>] pdflush+0xee/0x1b0
[<c01620b0>] wb_kupdate+0x0/0xf0
[<c01387c2>] kthread+0x42/0x70
[<c0138780>] kthread+0x0/0x70
[<c0103c1b>] kernel_thread_helper+0x7/0x1c
=======================
xfs_force_shutdown(md3,0x8) called from line 1165 of file fs/xfs/xfs_trans.c.
Return address = 0xc0385451
Filesystem "md3": Corruption of in-memory data detected. Shutting down
filesystem: md3
Please umount the filesystem, and rectify the problem(s)
Filesystem "md3": xfs_log_force: error 5 returned.
Filesystem "md3": xfs_log_force: error 5 returned.
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1af
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1af
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1af
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1ac
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1ac
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1ac
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1a9
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1a9
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: 1a9
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f4
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f4
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f5
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f5
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f6
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f6
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f7
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f7
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f8
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f8
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f9
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: f9
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fa
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fa
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fb
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fb
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fc
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fc
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fd
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fd
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fe
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: fe
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: ff
Filesystem "md3": Access to block zero in inode 537262150 start_block: 0
start_off: 0 blkcnt: 0 extent-state: 0 lastx: ff
Filesystem "md3": XFS internal error xfs_trans_cancel at line 1163 of file
fs/xfs/xfs_trans.c. Caller 0xc035db84
Pid: 18598, comm: pdflush Not tainted 2.6.26.3 #2
[<c0370119>] xfs_trans_cancel+0xe9/0x110
[<c035db84>] xfs_iomap_write_allocate+0x3a4/0x490
[<c035db84>] xfs_iomap_write_allocate+0x3a4/0x490
[<c035ec24>] xfs_iomap+0x334/0x410
[<c037af94>] xfs_map_blocks+0x44/0x90
[<c037c8df>] xfs_page_state_convert+0x53f/0x7a0
[<c037cc80>] xfs_vm_writepage+0x60/0x100
[<c0154cb8>] __writepage+0x8/0x30
[<c0155235>] write_cache_pages+0x215/0x300
[<c0154cb0>] __writepage+0x0/0x30
[<c0155340>] generic_writepages+0x20/0x30
[<c015537b>] do_writepages+0x2b/0x50
[<c0192c86>] __writeback_single_inode+0x86/0x310
[<c01208d7>] hrtick_set+0x67/0x110
[<c0155516>] get_dirty_limits+0x16/0x2c0
[<c01932de>] sync_sb_inodes+0x1ce/0x2b0
[<c0193741>] writeback_inodes+0x91/0xc0
[<c0155d23>] background_writeout+0x93/0xc0
[<c0156260>] pdflush+0x0/0x1b0
[<c015634e>] pdflush+0xee/0x1b0
[<c0155c90>] background_writeout+0x0/0xc0
[<c0137062>] kthread+0x42/0x70
[<c0137020>] kthread+0x0/0x70
[<c0103aa3>] kernel_thread_helper+0x7/0x14
=======================
xfs_force_shutdown(md3,0x8) called from line 1164 of file fs/xfs/xfs_trans.c.
Return address = 0xc0370131
Filesystem "md3": Corruption of in-memory data detected. Shutting down
filesystem: md3
Please umount the filesystem, and rectify the problem(s)
BUG: unable to handle kernel NULL pointer dereference at 00000000
IP: [<c037dbd9>] xfs_buf_delwri_split+0x59/0xf0
*pdpt = 00000000334b0001 *pde = 0000000000000000
Oops: 0000 [#1] SMP
Modules linked in: iscsi_tcp libiscsi scsi_transport_iscsi
Pid: 927, comm: xfsbufd Not tainted (2.6.26.3 #2)
EIP: 0060:[<c037dbd9>] EFLAGS: 00010282 CPU: 1
EIP is at xfs_buf_delwri_split+0x59/0xf0
EAX: 00000000 EBX: f5037cc0 ECX: 00000000 EDX: 00000000
ESI: ffffffdc EDI: f2501fbc EBP: f35003b0 ESP: f2501f98
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process xfsbufd (pid: 927, ti=f2500000 task=f30cd440 task.ti=f2500000)
Stack: 00001194 f35003b8 00000001 00000000 00001194 00000000 f2501fbc f3500380
c037ecf0 f2501fbc f2501fbc fffffffc f3500380 c037ec90 00000000 c0137062
c0137020 00000000 00000000 c0103aa3 f24ffdc0 00000000 00000000 00000000
Call Trace:
[<c037ecf0>] xfsbufd+0x60/0x100
[<c037ec90>] xfsbufd+0x0/0x100
[<c0137062>] kthread+0x42/0x70
[<c0137020>] kthread+0x0/0x70
[<c0103aa3>] kernel_thread_helper+0x7/0x14
=======================
Code: 7e e3 2e 00 8b 43 30 31 c9 8d 58 dc 39 c5 8b 53 24 89 4c 24 08 0f 84 7e
00 00 00 8d 72 dc eb 15 89 f6 ff 44 24 08 8d 46 24 39 c5 <8b> 56 24 74 69 89 f3
8d 72 dc 89 d8 e8 96 f1 ff ff 85 c0 75 e2
EIP: [<c037dbd9>] xfs_buf_delwri_split+0x59/0xf0 SS:ESP 0068:f2501f98
---[ end trace 6879b7e6cabe4008 ]---
Filesystem "md3": xfs_log_force: error 5 returned.
Filesystem "md3": xfs_log_force: error 5 returned.
|