xfs
[Top] [All Lists]

Re: suddenly slow writes on XFS Filesystem

To: "Stefan Priebe - Profihost AG" <s.priebe@xxxxxxxxxxxx>
Subject: Re: suddenly slow writes on XFS Filesystem
From: Martin Steigerwald <Martin@xxxxxxxxxxxx>
Date: Mon, 7 May 2012 16:32:03 +0200
Cc: xfs@xxxxxxxxxxx, Dave Chinner <david@xxxxxxxxxxxxx>, stan@xxxxxxxxxxxxxxxxx
In-reply-to: <4FA7D4D1.6070609@xxxxxxxxxxxx>
References: <4FA63DDA.9070707@xxxxxxxxxxxx> <201205071031.38856.Martin@xxxxxxxxxxxx> <4FA7D4D1.6070609@xxxxxxxxxxxx> (sfid-20120507_162145_568797_0C0B5263)
User-agent: KMail/1.13.7 (Linux/3.3.0-trunk-amd64; KDE/4.7.4; x86_64; ; )
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Am 07.05.2012 10:31, schrieb Martin Steigerwald:
> > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> > Did you verify that at the time you perceive slowness the servers you
> > backup can deliver data fast enough?
> 
> Yes. It works fine to another partition.

With more free space I suppose?

> 
> > I would like to now, whether there are really processes waiting for
> > I/O during rsync workload.
> > 
> > Can you try vmstat 5 and
> > while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
> 
> Here it is:
> # vmstat 5
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id wa
>  0  1      0 396688     48 7728136    0    0   229   374   22   32  1
> 12 85  2
>  2  0      0 421744     48 7717348    0    0  3722  1169 9015 8389  1 
> 2 95  2
>  0  1      0 405780     48 7751904    0    0  8937   207 10780 9290  2
> 2 88  7
>  0  0      0 486928     48 7733300    0    0  2526  1277 8275 7791  1 
> 2 96  1
>  0  0      0 416692     48 7778164    0    0  5046 43750 8548 8141  1 
> 2 95  2
>  0  0      0 444968     48 7777792    0    0  8021  1709 9315 8573  2 
> 2 94  2
>  2  0      0 357924     48 7946532    0    0 48181  1031 17646 12684 10
>  4 87  0
>  1  0      0 348696     48 8137200    0    0 74366  1056 24362 16775 15
>  5 81  0
>  1  0      0 391552     48 8279000    0    0 54693  1242 19224 13957 11
>  4 85  0

Hmm, there are some I/O wait times, but they are lower than I expected.

In the last lines the throughput is higher than in the first lines. Does 
that correlate to higher speed in rsync? It may also be excessive writes 
due to free space fragmentation, but Dave or someone else might be able to 
say whether such a huge difference can be blamed to free space 
fragmentation alone.

> # while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
> root     12493  2.0  0.2 101780 48392 ?        D    15:35   0:24 rsync
> --daemon
> root     13581  3.5  0.1  76832 20828 ?        D    15:50   0:10 rsync
> --daemon
> root     12493  2.0  0.2 101268 48508 ?        D    15:35   0:24 rsync
> --daemon
> root     12494  3.9  0.2 128220 44328 ?        D    15:35   0:47 rsync
> --daemon
> root     12493  2.0  0.2 101268 48508 ?        D    15:35   0:24 rsync
> --daemon

Still these rsyncs appear to be in uninteruptible sleep quite 
consistently. They shouldn´t be in that state when waiting on network 
socket readyness.

So this workloads seems to be I/O bound to me.

I suggest trying to keep the volume above 500 GB free and see whether that 
works consistently.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

<Prev in Thread] Current Thread [Next in Thread>