Quick note below, need one more bit of info.
On 3/14/2013 7:26 AM, Stan Hoeppner wrote:
> On 3/13/2013 11:37 PM, Dave Hall wrote:
>> If you'd rather I can re-post this to xfs@xxxxxxxxxxx, but I'm not clear
>> on exactly where this address leads. I am grateful for your response.
> No need, I'm CC'ing the list address. Read this entirely before hitting
>> So the details are that this is a 16 x 2GB 7200 rpm SATA drive array in
>> a RAID enclosure. The array is configured RAID6 (so 14 data spindles)
>> with a chunk size of 128k. The XFS formatted size is 26TB with 19TB
>> currently used.
> So your RAID6 stripe width is 14 * 128KB = 1,792KB.
>> The workload is a backup program called rsnapshot. If you're not
>> familiar, this program uses cp -al top create a linked copy of the
>> previous backup, and then rsync -av --del to copy in any changes. The
>> current snapshots contain about 14.8 million files. The total number of
>> snapshots is about 600.
> So you've got a metadata heavy workload with lots of links being created.
>> The performance problem that lead me to investigate XFS is that some
>> time around mid-November the cp -al step started running very long -
>> sometimes over 48 hours. Sometimes it runs in just a few hours. Prior
>> to then the entire backup consistenly finished in less than 12 hours.
>> When the cp -al is running long the output of dstat indicates that the
>> I/O to the fs is fairly light.
> The 'cp -al' command is a pure metadata workload, which means lots of
> writes to the filesystem directory trees, but not into files. And if
> your kernel is lower than 2.6.39 your log throughput would be pretty
> high as well. But given this is RAID6 you'll have significant RMW for
> these directory writes, maybe overwhelming RMW, driving latency up and
> thus actual bandwidth down. So dstat bytes throughput may be low, but
> %wa may be through the roof, making the dstat data you're watching
> completely misleading as to what's really going on, what's causing the
>> Please let me know if you need any further information.
> Yes, please provide the output of the following commands:
~$ uname -a
> ~$ grep xfs /etc/fstab
> ~$ xfs_info /dev/[mount-point]
> ~$ df /dev/[mount_point]
> ~$ df -i /dev/[mount_point]
> ~$ xfs_db -r -c freesp /dev/[mount-point]
> Also please provide the make/model of the RAID controller, the write
> cache size and if it is indeed enabled and working, as well as any
> errors, if any, logged by the controller in dmesg or elsewhere in Linux,
> or in the controller firmware.
>> Also, again, I
>> can post this to xfs@xxxxxxxxxxx but I'd really like to know more about
>> the address.
> Makes me where you obtained the list address. Apparently not from the
> official websites or you'd not have to ask. Maybe this will assuage
> your fears. ;)
> xfs@xxxxxxxxxxx is the official XFS mailing list submission address for
> the XFS developers and users. oss.sgi.com is the server provided and
> managed by SGI (www.sgi.com) that houses the XFS open source project.
> SGI created the XFS filesystem first released on their proprietary
> IRIX/MIPS computers in 1994. SGI open sourced XFS and ported it to
> Linux in the early 2000s.
> XFS is actively developed by a fairly large group of people, and AFAIK
> most of them are currently employed by Red Hat, including Dave Chinner,
> who also replied to your post. Dave wrote the delaylog code which will
> probably go a long way toward fixing your problem, if you're currently
> using 2.6.38 or lower and not mounting with this option enabled. It
> didn't become the default until 2.6.39.
> More info here http://www.xfs.org and here http://oss.sgi.com/projects/xfs/
> You bet.