Slow performance after ~4.5TB

[Dave Chinner]

On Mon, Nov 12, 2012 at 10:14:13AM +0200, Linas Jankauskas wrote:
> Hello,
> 
> we have 30 backup servers with 20TB backup partition each.
> While server is new and empty rsync is compying data prety fast, but
> when it reaches about 4.5TB write operation become very slow (about 10
> times slower).
> 
> I have attached cpu and disk graphs.
> 
> As you can see first week, while server was empty, rsync was using "user"
> cpu and data copying was fast. Later rsync started to use "system" cpu
> and data copying became much slower. Same situation is on all our backup
> servers. Before we had used smaller partition with ext4 and we had no
> problems.
> 
> Most time rsync is spending on ftruncate:
> 
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  99.99   18.362863      165431       111           ftruncate
>   0.00    0.000712           3       224       112 open
>   0.00    0.000195           1       257           write
>   0.00    0.000171           1       250           read
>   0.00    0.000075           1       112           lchown
>   0.00    0.000039           0       112           lstat
>   0.00    0.000028           0       112           close
>   0.00    0.000021           0       112           chmod
>   0.00    0.000011           0       396           select
>   0.00    0.000000           0       112           utimes
> ------ ----------- ----------- --------- --------- ----------------
> 100.00   18.364115                  1798       112 total

Never seen that before. More info needed. Start here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

And we can go from there.

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com