Hi,
Here is what I have done. I might have been stupic :|
/dev/md0 raid5 array 7+1 hot spare
# Sample raid-5 configuration
#
raiddev /dev/md0
raid-level 5
nr-raid-disks 7
# Spare disks for hot reconstruction
#nr-spare-disks 1
persistent-superblock 1
chunk-size 64
# the best one for maximum performance:
#
parity-algorithm left-symmetric
#parity-algorithm right-asymmetric
#parity-algorithm right-symmetric
device /dev/sdb1
raid-disk 0
device /dev/sdc1
raid-disk 1
device /dev/sdd1
raid-disk 2
device /dev/sde1
raid-disk 3
device /dev/sdf1
raid-disk 4
device /dev/sdg1
raid-disk 5
device /dev/sdh1
raid-disk 6
#
device /dev/sda1
spare-disk 0
#
1) remove the hot spare
#raidhotremove /dev/md0 /dev/sda1
<no problem>
2) put an xfs fs on the spare disk
# mkfs.xfs -f /dev/sda1
<no problem>
3) rsyncing from the xfs raid5 array (/dev/md0->/raid4)
to the new xfs partition (/dev/sda1 ->/sda1)
# cd /raid5
# rsync -av * /sda1
there is about ~55 MB of data
during the process rsync segfault :(
then for each of the subdir under /raid5 I did
one by one a
# rsync dir1 /sda1
# rsync dir2 /sda1
...
# rsync -av tftpboot /sda1
building file list ... done
tftpboot/
tftpboot/X86PC/
tftpboot/X86PC/UNDI/
tftpboot/X86PC/UNDI/BStrap/
tftpboot/X86PC/UNDI/BStrap/bstrap.0
tftpboot/X86PC/UNDI/linux-install/
tftpboot/X86PC/UNDI/linux-install/linux.0
Segmentation fault
rsync segfault again and I got a oops:
bash-2.04# ksymoops < /tmp/oops.txt
ksymoops 2.4.0 on i686 2.4.7-pre8-xfs. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.7-pre8-xfs/ (default)
-m /boot/System.map-2.4.7-pre8-xfs (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Error (expand_objects): cannot stat(/lib/3w-xxxx.o) for 3w-xxxx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/cdrom/driver.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/char/drm/drm.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/media/radio/radio.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/media/video/video.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/media/media.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/misc/misc.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/appletalk/appletalk.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/fc/fc.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/pcmcia/pcmcia_net.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/tokenring/tr.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/wan/wan.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/net/wireless/wireless_net.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/parport/driver.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/drivers/sound/sounddrivers.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/net/ipv4/netfilter/netfilter.o
Warning (read_object): no symbols in
/lib/modules/2.4.7-pre8-xfs/build/net/ipv6/netfilter/netfilter.o
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base
says c022d800, System.map says c01596f0. Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol nlmsvc_grace_period , lockd
says f8965234, /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o says
f8964664. Ignoring /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nlmsvc_ops , lockd says
f8965230, /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o says
f8964660. Ignoring /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nlmsvc_timeout , lockd says
f8965238, /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o says
f8964668. Ignoring /lib/modules/2.4.7-pre8-xfs/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nfs_debug , sunrpc says
f8953c60, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f8953920. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol nfsd_debug , sunrpc says
f8953c64, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f8953924. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol nlm_debug , sunrpc says
f8953c68, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f8953928. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_debug , sunrpc says
f8953c5c, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f895391c. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_garbage_args , sunrpc
says f8953c3c, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o
says f89538fc. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_success , sunrpc says
f8953c2c, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f89538ec. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol rpc_system_err , sunrpc says
f8953c40, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f8953900. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_one , sunrpc says
f8953c24, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f89538e4. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_two , sunrpc says
f8953c28, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f89538e8. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (compare_maps): mismatch on symbol xdr_zero , sunrpc says
f8953c20, /lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o says
f89538e0. Ignoring
/lib/modules/2.4.7-pre8-xfs/kernel/net/sunrpc/sunrpc.o entry
Warning (map_ksym_to_module): cannot match loaded module 3w-xxxx to a
unique module object. Trace may not be reliable.
Warning (map_ksym_to_module): cannot match loaded module sd_mod to a
unique module object. Trace may not be reliable.
Warning (map_ksym_to_module): cannot match loaded module scsi_mod to a
unique module object. Trace may not be reliable.
Jul 19 19:24:14 sheridan kernel: Unable to handle kernel paging request
at virtual address abdcee80
Jul 19 19:24:14 sheridan kernel: c023637c
Jul 19 19:24:14 sheridan kernel: *pde = 00000000
Jul 19 19:24:14 sheridan kernel: Oops: 0002
Jul 19 19:24:14 sheridan kernel: CPU: 1
Jul 19 19:24:14 sheridan kernel: EIP: 0010:[alloc_skb+332/456]
Jul 19 19:24:14 sheridan kernel: EIP: 0010:[<c023637c>]
Using defaults from ksymoops -t elf32-i386 -a i386
Jul 19 19:24:14 sheridan kernel: EFLAGS: 00010282
Jul 19 19:24:14 sheridan kernel: eax: abdcee80 ebx: ed5db480 ecx:
abdcc000 edx: dc216c80
Jul 19 19:24:14 sheridan kernel: esi: 000000f0 edi: 00002f20 ebp:
d3db6000 esp: d3db7eb4
Jul 19 19:24:14 sheridan kernel: ds: 0018 es: 0018 ss: 0018
Jul 19 19:24:14 sheridan kernel: Process rsync (pid: 8028,
stackpage=d3db7000)
Jul 19 19:24:14 sheridan kernel: Stack: d867fa40 00000000 00002e7c
00000002 00000000 c02358c1 00002e80 000000f0
Jul 19 19:24:14 sheridan kernel: d864ceec 00002e7c d67c7134
d67c70e0 d3db6000 c0279ab2 d867fa40 00002e7c
Jul 19 19:24:14 sheridan kernel: 00000040 d3db7f1c d864ceec
d3db7f84 c02799a0 d864ceec c0130fc4 00000000
Jul 19 19:24:14 sheridan kernel: Call Trace:
[sock_alloc_send_skb+113/264] [unix_stream_sendmsg+274/828]
[unix_stream_sendmsg+0/828] [free_pages+36/40] [sock_sendmsg+129/164]
[unix_stream_sendmsg+0/828] [sock_write+163/172]
Jul 19 19:24:14 sheridan kernel: Call Trace: [<c02358c1>] [<c0279ab2>]
[<c02799a0>] [<c0130fc4>] [<c023315d>] [<c02799a0>] [<c023337b>]
Jul 19 19:24:14 sheridan kernel: [<c013768a>] [<c0106fcb>]
Jul 19 19:24:14 sheridan kernel: Code: c7 00 01 00 00 00 8b 83 88 00 00
00 c7 40 04 00 00 00 00 8b
>>EIP; c023637c <alloc_skb+14c/1c8> <=====
Trace; c02358c1 <sock_alloc_send_skb+71/108>
Trace; c0279ab2 <unix_stream_sendmsg+112/33c>
Trace; c02799a0 <unix_stream_sendmsg+0/33c>
Trace; c0130fc4 <free_pages+24/28>
Trace; c023315d <sock_sendmsg+81/a4>
Trace; c02799a0 <unix_stream_sendmsg+0/33c>
Trace; c023337b <sock_write+a3/ac>
Trace; c013768a <sys_write+8e/c4>
Trace; c0106fcb <system_call+33/38>
Code; c023637c <alloc_skb+14c/1c8>
00000000 <_EIP>:
Code; c023637c <alloc_skb+14c/1c8> <=====
0: c7 00 01 00 00 00 movl $0x1,(%eax) <=====
Code; c0236382 <alloc_skb+152/1c8>
6: 8b 83 88 00 00 00 mov 0x88(%ebx),%eax
Code; c0236388 <alloc_skb+158/1c8>
c: c7 40 04 00 00 00 00 movl $0x0,0x4(%eax)
Code; c023638f <alloc_skb+15f/1c8>
13: 8b 00 mov (%eax),%eax
Jul 19 19:45:10 sheridan kernel: Unable to handle kernel paging request
at virtual address 3d537020
Jul 19 19:45:10 sheridan kernel: c023637c
Jul 19 19:45:10 sheridan kernel: *pde = 00000000
Jul 19 19:45:10 sheridan kernel: Oops: 0002
Jul 19 19:45:10 sheridan kernel: CPU: 0
Jul 19 19:45:10 sheridan kernel: EIP: 0010:[alloc_skb+332/456]
Jul 19 19:45:10 sheridan kernel: EIP: 0010:[<c023637c>]
Jul 19 19:45:10 sheridan kernel: EFLAGS: 00010202
Jul 19 19:45:10 sheridan kernel: eax: 3d537020 ebx: f200e320 ecx:
3d534000 edx: dc216c80
Jul 19 19:45:10 sheridan kernel: esi: 000000f0 edi: 000030c0 ebp:
f604c000 esp: f604deb4
Jul 19 19:45:10 sheridan kernel: ds: 0018 es: 0018 ss: 0018
Jul 19 19:45:10 sheridan kernel: Process rsync (pid: 8102,
stackpage=f604d000)
Jul 19 19:45:10 sheridan kernel: Stack: f74140e0 00000000 00003008
00000002 00000202 c02358c1 00003020 000000f0
Jul 19 19:45:10 sheridan kernel: eda6954c 00003008 f751f5d4
f751f580 f604c000 c0279ab2 f74140e0 00003008
Jul 19 19:45:10 sheridan kernel: 00000040 f604df1c eda6954c
f604df84 c02799a0 eda6954c c0130fc4 00000000
Jul 19 19:45:10 sheridan kernel: Call Trace:
[sock_alloc_send_skb+113/264] [unix_stream_sendmsg+274/828]
[unix_stream_sendmsg+0/828] [free_pages+36/40] [sock_sendmsg+129/164]
[unix_stream_sendmsg+0/828] [sock_write+163/172]
Jul 19 19:45:10 sheridan kernel: Call Trace: [<c02358c1>] [<c0279ab2>]
[<c02799a0>] [<c0130fc4>] [<c023315d>] [<c02799a0>] [<c023337b>]
Jul 19 19:45:10 sheridan kernel: [<c013768a>] [<c0106fcb>]
Jul 19 19:45:10 sheridan kernel: Code: c7 00 01 00 00 00 8b 83 88 00 00
00 c7 40 04 00 00 00 00 8b
>>EIP; c023637c <alloc_skb+14c/1c8> <=====
Trace; c02358c1 <sock_alloc_send_skb+71/108>
Trace; c0279ab2 <unix_stream_sendmsg+112/33c>
Trace; c02799a0 <unix_stream_sendmsg+0/33c>
Trace; c0130fc4 <free_pages+24/28>
Trace; c023315d <sock_sendmsg+81/a4>
Trace; c02799a0 <unix_stream_sendmsg+0/33c>
Trace; c023337b <sock_write+a3/ac>
Trace; c013768a <sys_write+8e/c4>
Trace; c0106fcb <system_call+33/38>
Code; c023637c <alloc_skb+14c/1c8>
00000000 <_EIP>:
Code; c023637c <alloc_skb+14c/1c8> <=====
0: c7 00 01 00 00 00 movl $0x1,(%eax) <=====
Code; c0236382 <alloc_skb+152/1c8>
6: 8b 83 88 00 00 00 mov 0x88(%ebx),%eax
Code; c0236388 <alloc_skb+158/1c8>
c: c7 40 04 00 00 00 00 movl $0x0,0x4(%eax)
Code; c023638f <alloc_skb+15f/1c8>
13: 8b 00 mov (%eax),%eax
34 warnings and 3 errors issued. Results may not be reliable.
4) I umount both filesystems and run xfs_repair -n on them
#xfs_repair -n /dev/md0
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
<...>
- agno = 107
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
<...>
- agno = 107
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
- traversal finished ...
- traversing all unattached subtrees ...
- traversals finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
# xfs_repair -n /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
<...>
- agno = 17
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
<...>
- agno = 17
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem starting at / ...
- traversal finished ...
- traversing all unattached subtrees ...
- traversals finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
5) I could remount the plain xfs partition but not the raid5 array:(
# mount /raid5/
mount: wrong fs type, bad option, bad superblock on /dev/md0,
or too many mounted file systems
on the syslog file:
Jul 19 20:16:56 nfs kernel: XFS: xlog_recover_reorder_trans:
unrecognized type of log
operation
Any idea what are the next steps?
Regards,
Tru
--
Dr Tru Huynh | Bioinformatique Structurale
mailto:tru@xxxxxxxxxx | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France
|