<div dir="ltr">Package: xfsprogs<br>Version: 3.1.4<br>Justification: breaks unrelated software<br>Severity: critical<br>
Tags: upstream<br><br>*** Please type your report below this line ***<br><br>As part of my standard virtual machine template for MySQL, I created an LVM setup as follows:<br>pvcreate /dev/sdb<br>vgcreate vg0 /dev/sdb<br>
lvcreate -n mysql -L 10G vg0<br>
lvcreate -n mysqlbinlog -L 5G vg0<br>lvcreate -n mysqllog -L 5G vg0<br><br>Then, after I deploy the template, I resize the LVs and xfs_growfs them, apt-get upgrade, and restart.<br><br>I also frequently (multiple times per week) extend MySQL data volumes on the fly using<br>
echo 1 > /sys/block/sdb/device/rescane<br>pvresize /dev/sdb<br>lvresize -l +100%FREE /dev/vg0/mysql<br>xfs_growfs /mysql<br><br>This
has worked hundreds of times. Twice, though, on Debian 6.x (I don't
have enough 7.x boxes in production yet) I've had something happen with
XFS. Once, after an extend, it was unmountable. I assumed this was an
error on my part, created the FS, and moved on. <br>
<br>Today, I got an SQL error on a MySQL replication box:<br><br> Error 'The table 'XXXXX' is full' on query. Default database: XXXXXYYYYYY<br><br>Normally
this would be a configuration issue in MySQL related to a setting
called "innodb_data_file_path". Mine is configured correctly; otherwise
all of my templates would have died years ago. On this particular
host, I extended /mysql (the data volume) to 120G before reboot. Today,
when it hit what appears to be the *original* size:<br>
<br>/dev/mapper/vg0-mysql <div dir="ltr"> 125814784 10473148 115341636 9% /mysql<br><br>XFS threw allocation errors.<br><br>[4051547.986869] ffff880638219200: 58 41 47 46 00 00 00 01 00 00 00 0c 00 05 00 00 XAGF............<br>
[4051547.987636] Filesystem "dm-2": XFS internal error xfs_alloc_read_agf at line 2293 of file /build/buildd-linux-2.6_2.6.32-48squeeze1-amd64-qu4MIV/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/xfs/xfs_alloc.c. Caller 0xffffffffa021fcb0<br>
[4051547.987639] <br>[4051547.988226] Pid: 3367, comm: mysqld Not tainted 2.6.32-5-amd64 #1<br>[4051547.988228] Call Trace:<br>[4051547.988287] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4051547.988298] [<ffffffffa021fc73>] ? xfs_read_agf+0x12e/0x149 [xfs]<br>
[4051547.988307] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4051547.988317] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4051547.988331] [<ffffffffa021fd45>] ? xfs_alloc_pagf_init+0x13/0x34 [xfs]<br>
[4051547.988342] [<ffffffffa022c3f1>] ? xfs_inode_is_filestream+0x24/0x51 [xfs]<br>[4051547.988353] [<ffffffffa022c67d>] ? xfs_bmap_btalloc+0x25f/0x732 [xfs]<br>[4051547.988368] [<ffffffffa0228b70>] ? xfs_bmap_search_extents+0x57/0xd1 [xfs]<br>
[4051547.988379] [<ffffffffa022ee83>] ? xfs_bmapi+0x876/0x104d [xfs]<br>[4051547.988386] [<ffffffff8118eec3>] ? _atomic_dec_and_lock+0x33/0x50<br>[4051547.988406] [<ffffffffa0257217>] ? xfs_trans_reserve+0xc8/0x19d [xfs]<br>
[4051547.988419] [<ffffffffa024a51f>] ? xfs_iomap_write_direct+0x305/0x3f5 [xfs]<br>[4051547.988432] [<ffffffffa024a7bc>] ? xfs_iomap+0x1ad/0x285 [xfs]<br>[4051547.988444] [<ffffffffa025db53>] ? __xfs_get_blocks+0x6c/0x15c [xfs]<br>
[4051547.988450] [<ffffffff81062311>] ? queue_work_on+0x45/0x4f<br>[4051547.988455] [<ffffffff8111442a>] ? __blockdev_direct_IO+0x575/0xa60<br>[4051547.988461] [<ffffffff810482ef>] ? finish_task_switch+0x3a/0xaf<br>
[4051547.988472] [<ffffffffa025d983>] ? xfs_vm_direct_IO+0xd4/0x102 [xfs]<br>[4051547.988483] [<ffffffffa025dc43>] ? xfs_get_blocks_direct+0x0/0x11 [xfs]<br>[4051547.988494] [<ffffffffa025d791>] ? xfs_end_io_direct+0x0/0x5f [xfs]<br>
[4051547.988499] [<ffffffff810b55d6>] ? generic_file_direct_write+0xdd/0x152<br>[4051547.988510] [<ffffffffa0264532>] ? xfs_write+0x44f/0x6ea [xfs]<br>[4051547.988523] [<ffffffffa02442ca>] ? xfs_iunlock+0x31/0x7c [xfs]<br>
[4051547.988528] [<ffffffff810ef21e>] ? do_sync_write+0xce/0x113<br>[4051547.988532] [<ffffffff810652b2>] ? autoremove_wake_function+0x0/0x2e<br>[4051547.988535] [<ffffffff810482ef>] ? finish_task_switch+0x3a/0xaf<br>
[4051547.988538] [<ffffffff810efb70>] ? vfs_write+0xa9/0x102<br>[4051547.988541] [<ffffffff810efc20>] ? sys_pwrite64+0x57/0x77<br>[4051547.988545] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b<br>
[4053278.621482] ffff880638219200: 58 41 47 46 00 00 00 01 00 00 00 0c 00 05 00 00 XAGF............<br>[4053278.621761] Filesystem "dm-2": XFS internal error xfs_alloc_read_agf at line 2293 of file /build/buildd-linux-2.6_2.6.32-48squeeze1-amd64-qu4MIV/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/xfs/xfs_alloc.c. Caller 0xffffffffa021fcb0<br>
[4053278.621764] <br>[4053278.622352] Pid: 3367, comm: mysqld Not tainted 2.6.32-5-amd64 #1<br>[4053278.622354] Call Trace:<br>[4053278.622381] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4053278.622392] [<ffffffffa021fc73>] ? xfs_read_agf+0x12e/0x149 [xfs]<br>
[4053278.622401] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4053278.622411] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4053278.622421] [<ffffffffa021fd45>] ? xfs_alloc_pagf_init+0x13/0x34 [xfs]<br>
[4053278.622432] [<ffffffffa022c3f1>] ? xfs_inode_is_filestream+0x24/0x51 [xfs]<br>[4053278.622442] [<ffffffffa022c67d>] ? xfs_bmap_btalloc+0x25f/0x732 [xfs]<br>[4053278.622453] [<ffffffffa0228b70>] ? xfs_bmap_search_extents+0x57/0xd1 [xfs]<br>
[4053278.622465] [<ffffffffa022ee83>] ? xfs_bmapi+0x876/0x104d [xfs]<br>[4053278.622486] [<ffffffff8118eec3>] ? _atomic_dec_and_lock+0x33/0x50<br>[4053278.622501] [<ffffffffa0257217>] ? xfs_trans_reserve+0xc8/0x19d [xfs]<br>
[4053278.622514] [<ffffffffa024a51f>] ? xfs_iomap_write_direct+0x305/0x3f5 [xfs]<br>[4053278.622527] [<ffffffffa024a7bc>] ? xfs_iomap+0x1ad/0x285 [xfs]<br>[4053278.622538] [<ffffffffa025db53>] ? __xfs_get_blocks+0x6c/0x15c [xfs]<br>
[4053278.622551] [<ffffffff8111442a>] ? __blockdev_direct_IO+0x575/0xa60<br>[4053278.622563] [<ffffffffa025d983>] ? xfs_vm_direct_IO+0xd4/0x102 [xfs]<br>[4053278.622574] [<ffffffffa025dc43>] ? xfs_get_blocks_direct+0x0/0x11 [xfs]<br>
[4053278.622585] [<ffffffffa025d791>] ? xfs_end_io_direct+0x0/0x5f [xfs]<br>[4053278.622593] [<ffffffff810b55d6>] ? generic_file_direct_write+0xdd/0x152<br>[4053278.622605] [<ffffffffa0264532>] ? xfs_write+0x44f/0x6ea [xfs]<br>
[4053278.622618] [<ffffffffa02442ca>] ? xfs_iunlock+0x31/0x7c [xfs]<br>[4053278.622625] [<ffffffff810ef21e>] ? do_sync_write+0xce/0x113<br>[4053278.622637] [<ffffffff810652b2>] ? autoremove_wake_function+0x0/0x2e<br>
[4053278.622646] [<ffffffff810482ef>] ? finish_task_switch+0x3a/0xaf<br>[4053278.622650] [<ffffffff810efb70>] ? vfs_write+0xa9/0x102<br>[4053278.622652] [<ffffffff810efc20>] ? sys_pwrite64+0x57/0x77<br>
[4053278.622660] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b<br>[4055487.227132] ffff880638219200: 58 41 47 46 00 00 00 01 00 00 00 0c 00 05 00 00 XAGF............<br>[4055487.227678] Filesystem "dm-2": XFS internal error xfs_alloc_read_agf at line 2293 of file /build/buildd-linux-2.6_2.6.32-48squeeze1-amd64-qu4MIV/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/xfs/xfs_alloc.c. Caller 0xffffffffa021fcb0<br>
[4055487.227681] <br>[4055487.228267] Pid: 32572, comm: mysqld Not tainted 2.6.32-5-amd64 #1<br>[4055487.228269] Call Trace:<br>[4055487.228292] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4055487.228302] [<ffffffffa021fc73>] ? xfs_read_agf+0x12e/0x149 [xfs]<br>
[4055487.228311] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4055487.228321] [<ffffffffa021fcb0>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs]<br>[4055487.228330] [<ffffffffa021fd45>] ? xfs_alloc_pagf_init+0x13/0x34 [xfs]<br>
[4055487.228341] [<ffffffffa022c3f1>] ? xfs_inode_is_filestream+0x24/0x51 [xfs]<br>[4055487.228352] [<ffffffffa022c67d>] ? xfs_bmap_btalloc+0x25f/0x732 [xfs]<br>[4055487.228363] [<ffffffffa0228b70>] ? xfs_bmap_search_extents+0x57/0xd1 [xfs]<br>
[4055487.228374] [<ffffffffa022ee83>] ? xfs_bmapi+0x876/0x104d [xfs]<br>[4055487.228388] [<ffffffffa0257217>] ? xfs_trans_reserve+0xc8/0x19d [xfs]<br>[4055487.228400] [<ffffffffa024a51f>] ? xfs_iomap_write_direct+0x305/0x3f5 [xfs]<br>
[4055487.228413] [<ffffffffa024a7bc>] ? xfs_iomap+0x1ad/0x285 [xfs]<br>[4055487.228425] [<ffffffffa025db53>] ? __xfs_get_blocks+0x6c/0x15c [xfs]<br>[4055487.228431] [<ffffffff8111442a>] ? __blockdev_direct_IO+0x575/0xa60<br>
[4055487.228442] [<ffffffffa025d983>] ? xfs_vm_direct_IO+0xd4/0x102 [xfs]<br>[4055487.228453] [<ffffffffa025dc43>] ? xfs_get_blocks_direct+0x0/0x11 [xfs]<br>[4055487.228464] [<ffffffffa025d791>] ? xfs_end_io_direct+0x0/0x5f [xfs]<br>
[4055487.228468] [<ffffffff810b55d6>] ? generic_file_direct_write+0xdd/0x152<br>[4055487.228479] [<ffffffffa0264532>] ? xfs_write+0x44f/0x6ea [xfs]<br>[4055487.228492] [<ffffffffa02442ca>] ? xfs_iunlock+0x31/0x7c [xfs]<br>
[4055487.228496] [<ffffffff810ef21e>] ? do_sync_write+0xce/0x113<br>[4055487.228500] [<ffffffff810652b2>] ? autoremove_wake_function+0x0/0x2e<br>[4055487.228503] [<ffffffff810efb70>] ? vfs_write+0xa9/0x102<br>
[4055487.228505] [<ffffffff810efc20>] ? sys_pwrite64+0x57/0x77<br>[4055487.228509] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b<br><br>Throwing a dd at the FS works fine:<br><br>dd if=/dev/urandom of=/mysql/data/somefile.out bs=1M count=512<br>
<br>Two things might be going on here that are special<br><br>1) I'm
using a pretty customized MySQL configuration, including a setting
called O_DIRECT, which turns on direct IO for InnoDB flushes.
Therefore, the code path for some InnoDB writes is going to be different
from what dd does.<br>
2) This is SAN-backed storage; it shouldn't matter, but it does introduce IO latency.<br><br><br>I've
been trying to find a way to fix this "hot"; these are production
systems. Luckily it was just a read-only replica. If this had happened
to the master, there would probably have been data corruption (hence the
sev 1 ticket). Couldn't find a hot fix, so I'm going to unmount/repair
and then reboot. I'll keep track of what happens so I can attach it to
this report later if possible.<br>
<br>Thanks!<br><br>Wes<br><br>-- System Information:<br>Debian Release: 6.0.7<br> APT prefers oldstable-updates<br> APT policy: (500, 'oldstable-updates'), (500, 'oldstable')<br>Architecture: amd64 (x86_64)<br>
<br>Kernel: Linux 2.6.32-5-amd64 (SMP w/4 CPU cores)<br>Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)<br>Shell: /bin/sh linked to /bin/bash<br><br>Versions of packages xfsprogs depends on:<br>ii libblkid1 2.17.2-9 block device id library<br>
ii libc6 2.11.3-4 Embedded GNU C Library: Shared lib<br>ii libreadline5 5.2-7 GNU readline and history libraries<br>ii libuuid1 2.17.2-9 Universally Unique ID library<br>
<br>xfsprogs recommends no packages.<br><br>Versions of packages xfsprogs suggests:<br>pn acl <none> (no description available)<br>pn attr <none> (no description available)<br>
pn quota <none> (no description available)<br>pn xfsdump <none> (no description available)<br><br>-- no debconf information</div></div>