[Top] [All Lists]

Re: gather write metrics on multiple files

To: xfs@xxxxxxxxxxx
Subject: Re: gather write metrics on multiple files
From: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Thu, 09 Oct 2014 00:24:20 -0500
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <543613E7.70508@xxxxxxxxx>
References: <543611CF.6030904@xxxxxxxxxxxxxxxxx> <543613E7.70508@xxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Icedove/24.7.0
On 10/08/2014 11:49 PM, Joe Landman wrote:
> On 10/09/2014 12:40 AM, Stan Hoeppner wrote:
>> Does anyone know of a utility that can track writes to files in an
>> XFS directory tree, or filesystem wide for that matter, and gather
>> filesystem blocks written per second data, or simply KiB/s, etc?  I
>> need to analyze an application's actual IO behavior to see if it
>> matches what I'm being told the application is supposed to be doing.
> We've written a few for this purpose (local IO probing).
> Start with collectl (looks at /proc/diskstats), and others.  Our tools go to 
> /proc/diskstats, and use this to compute BW and IOPs per device.
> If you need to log it for a long time, set up a time series database (we use 
> influxdb and the graphite plugin).  Then grab your favorite metrics tool that 
> talks to graphite/influxdb (I like https://github.com/joelandman/sios-metrics 
> for obvious reasons), and start collecting data.

I'm told we have 800 threads writing to nearly as many files concurrently on a 
single XFS on a 12+2 spindle RAID6 LUN.  Achieved data rate is currently ~300 
MiB/s.  Some of these are files are supposedly being written at a rate of only 
32KiB every 2-3 seconds, while some (two) are ~50 MiB/s.  I need to determine 
how many bytes we're writing to each of the low rate files, and how many files, 
to figure out RMW mitigation strategies.  Out of the apparent 800 streams 700 
are these low data rate suckers, one stream writing per file.  

Nary a stock RAID controller is going to be able to assemble full stripes out 
of these small slow writes.  With a 768 KiB stripe that's what, 24 seconds to 
fill it at 2 seconds per 32 KiB IO?  I've been playing with bcache for a few 
days but it actually drops throughput by about 30% no matter how I turn its 
knobs.  Unless I can get Kent to respond to some of my questions bcache will be 
a dead end.  I had high hopes for it, thinking it would turn these small random 
IOs into larger sequential writes.  It may actually be doing so, but it's doing 
something else too, and badly.  IO times go through the roof once bcache starts 
gobbling IOs, and throughput to the LUNs drops significantly even though bcache 
is writing 50-100 MIB/s to the SSD.  Not sure what's causing that.

>> Thanks, Stan
>> _______________________________________________ xfs mailing list
>> xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs

<Prev in Thread] Current Thread [Next in Thread>