xfs
[Top] [All Lists]

Re: Poor performance using discard

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: Poor performance using discard
From: Thomas Lynema <lyz27@xxxxxxxxx>
Date: Fri, 02 Mar 2012 10:41:50 -0500
Cc: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1330702918; bh=8/B/wapUP2xbeDd1WmNWPA1jxDSc0xO+EaJv5XFtAbc=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References:Content-Type:X-Mailer:Mime-Version; b=KnZYXtdtA9bwpN7obo/0FQw5ssmEmCvIMkJPaTPz/fCLT1dMyEzYGfZa5ibOEJNl/GR2mGNNoCekgZ8vX/0TZViOCNV/zMQUKXlp5bSTVJZBcvkE2wZOCttzHOtP8Ync3uZc413A3cVKDzkyKZvxkb7yaqTzLGJ68/wF/0cOydk=
In-reply-to: <20120229040819.GZ3592@dastard>
References: <1330469778.9688.7.camel@core24> <20120229012259.GW3592@dastard> <1330480826.9688.23.camel@core24> <20120229040819.GZ3592@dastard>
[....]

> Any errors in dmesg?
None
> Also, I think you need to provide a block trace (output of
> blktrace/blkparse for the rm -rf workloads) for both the XFS and
> ext4 cases

Here's the output from running similar
commands as Christoph:

blkparse xfs.trace.blktrace.* | grep -w D | wc -l 
1134996
blkparse ext4.trace.blktrace.* | grep -w D | wc -l 
27240

Attached archives of half of the ext4 results. 

Summary:

CPU0 (ext4.trace):
 Reads Queued:         372,    1,944KiB  Writes Queued:       1,932,    7,728KiB
 Read Dispatches:      372,    1,944KiB  Write Dispatches:       19,    7,756KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      419,    2,104KiB  Writes Completed:       13,    5,123KiB
 Read Merges:            0,        0KiB  Write Merges:        1,914,    7,656KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            12               Timer unplugs:           0
CPU1 (ext4.trace):
 Reads Queued:       3,119,   12,484KiB  Writes Queued:       2,591,  420,624KiB
 Read Dispatches:    3,168,   12,484KiB  Write Dispatches:      356,  423,100KiB
 Reads Requeued:        49               Writes Requeued:       115
 Reads Completed:    3,271,   13,104KiB  Writes Completed:      251,   11,317KiB
 Read Merges:            0,        0KiB  Write Merges:        2,358,    9,504KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            10               Timer unplugs:           2
CPU2 (ext4.trace):
 Reads Queued:         191,      844KiB  Writes Queued:           2,        3KiB
 Read Dispatches:      191,      844KiB  Write Dispatches:        1,        3KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      144,      684KiB  Writes Completed:       10,    2,608KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:            29               Write depth:            32
 IO unplugs:             6               Timer unplugs:           0
CPU3 (ext4.trace):
 Reads Queued:       1,273,    5,116KiB  Writes Queued:         852,  112,049KiB
 Read Dispatches:    1,315,    5,116KiB  Write Dispatches:       72,  109,545KiB
 Reads Requeued:        42               Writes Requeued:        11
 Reads Completed:    1,121,    4,496KiB  Writes Completed:       56,    4,024KiB
 Read Merges:            0,        0KiB  Write Merges:          779,    3,392KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            15               Timer unplugs:          11

Total (ext4.trace):
 Reads Queued:       4,955,   20,388KiB  Writes Queued:       5,377,  540,404KiB
 Read Dispatches:    5,046,   20,388KiB  Write Dispatches:      448,  540,404KiB
 Reads Requeued:        91               Writes Requeued:       126
 Reads Completed:    4,955,   20,388KiB  Writes Completed:      330,   23,072KiB
 Read Merges:            0,        0KiB  Write Merges:        5,051,   20,552KiB
 IO unplugs:            43               Timer unplugs:          13

Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 -   0.0%)

CPU0 (ext4.trace):
 Reads Queued:         372,    1,944KiB  Writes Queued:       1,932,    7,728KiB
 Read Dispatches:      372,    1,944KiB  Write Dispatches:       19,    7,756KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      419,    2,104KiB  Writes Completed:       13,    5,123KiB
 Read Merges:            0,        0KiB  Write Merges:        1,914,    7,656KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            12               Timer unplugs:           0
CPU1 (ext4.trace):
 Reads Queued:       3,119,   12,484KiB  Writes Queued:       2,591,  420,624KiB
 Read Dispatches:    3,168,   12,484KiB  Write Dispatches:      356,  423,100KiB
 Reads Requeued:        49               Writes Requeued:       115
 Reads Completed:    3,271,   13,104KiB  Writes Completed:      251,   11,317KiB
 Read Merges:            0,        0KiB  Write Merges:        2,358,    9,504KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            10               Timer unplugs:           2
CPU2 (ext4.trace):
 Reads Queued:         191,      844KiB  Writes Queued:           2,        3KiB
 Read Dispatches:      191,      844KiB  Write Dispatches:        1,        3KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      144,      684KiB  Writes Completed:       10,    2,608KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:            29               Write depth:            32
 IO unplugs:             6               Timer unplugs:           0
CPU3 (ext4.trace):
 Reads Queued:       1,273,    5,116KiB  Writes Queued:         852,  112,049KiB
 Read Dispatches:    1,315,    5,116KiB  Write Dispatches:       72,  109,545KiB
 Reads Requeued:        42               Writes Requeued:        11
 Reads Completed:    1,121,    4,496KiB  Writes Completed:       56,    4,024KiB
 Read Merges:            0,        0KiB  Write Merges:          779,    3,392KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            15               Timer unplugs:          11

Total (ext4.trace):
 Reads Queued:       4,955,   20,388KiB  Writes Queued:       5,377,  540,404KiB
 Read Dispatches:    5,046,   20,388KiB  Write Dispatches:      448,  540,404KiB
 Reads Requeued:        91               Writes Requeued:       126
 Reads Completed:    4,955,   20,388KiB  Writes Completed:      330,   23,072KiB
 Read Merges:            0,        0KiB  Write Merges:        5,051,   20,552KiB
 IO unplugs:            43               Timer unplugs:          13

Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 -   0.0%)

CPU0 (ext4.trace):
 Reads Queued:         372,    1,944KiB  Writes Queued:       1,932,    7,728KiB
 Read Dispatches:      372,    1,944KiB  Write Dispatches:       19,    7,756KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      419,    2,104KiB  Writes Completed:       13,    5,123KiB
 Read Merges:            0,        0KiB  Write Merges:        1,914,    7,656KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            12               Timer unplugs:           0
CPU1 (ext4.trace):
 Reads Queued:       3,119,   12,484KiB  Writes Queued:       2,591,  420,624KiB
 Read Dispatches:    3,168,   12,484KiB  Write Dispatches:      356,  423,100KiB
 Reads Requeued:        49               Writes Requeued:       115
 Reads Completed:    3,271,   13,104KiB  Writes Completed:      251,   11,317KiB
 Read Merges:            0,        0KiB  Write Merges:        2,358,    9,504KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            10               Timer unplugs:           2
CPU2 (ext4.trace):
 Reads Queued:         191,      844KiB  Writes Queued:           2,        3KiB
 Read Dispatches:      191,      844KiB  Write Dispatches:        1,        3KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      144,      684KiB  Writes Completed:       10,    2,608KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:            29               Write depth:            32
 IO unplugs:             6               Timer unplugs:           0
CPU3 (ext4.trace):
 Reads Queued:       1,273,    5,116KiB  Writes Queued:         852,  112,049KiB
 Read Dispatches:    1,315,    5,116KiB  Write Dispatches:       72,  109,545KiB
 Reads Requeued:        42               Writes Requeued:        11
 Reads Completed:    1,121,    4,496KiB  Writes Completed:       56,    4,024KiB
 Read Merges:            0,        0KiB  Write Merges:          779,    3,392KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            15               Timer unplugs:          11

Total (ext4.trace):
 Reads Queued:       4,955,   20,388KiB  Writes Queued:       5,377,  540,404KiB
 Read Dispatches:    5,046,   20,388KiB  Write Dispatches:      448,  540,404KiB
 Reads Requeued:        91               Writes Requeued:       126
 Reads Completed:    4,955,   20,388KiB  Writes Completed:      330,   23,072KiB
 Read Merges:            0,        0KiB  Write Merges:        5,051,   20,552KiB
 IO unplugs:            43               Timer unplugs:          13

Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 -   0.0%)

CPU0 (ext4.trace):
 Reads Queued:         372,    1,944KiB  Writes Queued:       1,932,    7,728KiB
 Read Dispatches:      372,    1,944KiB  Write Dispatches:       19,    7,756KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      419,    2,104KiB  Writes Completed:       13,    5,123KiB
 Read Merges:            0,        0KiB  Write Merges:        1,914,    7,656KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            12               Timer unplugs:           0
CPU1 (ext4.trace):
 Reads Queued:       3,119,   12,484KiB  Writes Queued:       2,591,  420,624KiB
 Read Dispatches:    3,168,   12,484KiB  Write Dispatches:      356,  423,100KiB
 Reads Requeued:        49               Writes Requeued:       115
 Reads Completed:    3,271,   13,104KiB  Writes Completed:      251,   11,317KiB
 Read Merges:            0,        0KiB  Write Merges:        2,358,    9,504KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            10               Timer unplugs:           2
CPU2 (ext4.trace):
 Reads Queued:         191,      844KiB  Writes Queued:           2,        3KiB
 Read Dispatches:      191,      844KiB  Write Dispatches:        1,        3KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      144,      684KiB  Writes Completed:       10,    2,608KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:            29               Write depth:            32
 IO unplugs:             6               Timer unplugs:           0
CPU3 (ext4.trace):
 Reads Queued:       1,273,    5,116KiB  Writes Queued:         852,  112,049KiB
 Read Dispatches:    1,315,    5,116KiB  Write Dispatches:       72,  109,545KiB
 Reads Requeued:        42               Writes Requeued:        11
 Reads Completed:    1,121,    4,496KiB  Writes Completed:       56,    4,024KiB
 Read Merges:            0,        0KiB  Write Merges:          779,    3,392KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            15               Timer unplugs:          11

Total (ext4.trace):
 Reads Queued:       4,955,   20,388KiB  Writes Queued:       5,377,  540,404KiB
 Read Dispatches:    5,046,   20,388KiB  Write Dispatches:      448,  540,404KiB
 Reads Requeued:        91               Writes Requeued:       126
 Reads Completed:    4,955,   20,388KiB  Writes Completed:      330,   23,072KiB
 Read Merges:            0,        0KiB  Write Merges:        5,051,   20,552KiB
 IO unplugs:            43               Timer unplugs:          13

Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 -   0.0%)

CPU0 (ext4.trace):
 Reads Queued:         372,    1,944KiB  Writes Queued:       1,932,    7,728KiB
 Read Dispatches:      372,    1,944KiB  Write Dispatches:       19,    7,756KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      419,    2,104KiB  Writes Completed:       13,    5,123KiB
 Read Merges:            0,        0KiB  Write Merges:        1,914,    7,656KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            12               Timer unplugs:           0
CPU1 (ext4.trace):
 Reads Queued:       3,119,   12,484KiB  Writes Queued:       2,591,  420,624KiB
 Read Dispatches:    3,168,   12,484KiB  Write Dispatches:      356,  423,100KiB
 Reads Requeued:        49               Writes Requeued:       115
 Reads Completed:    3,271,   13,104KiB  Writes Completed:      251,   11,317KiB
 Read Merges:            0,        0KiB  Write Merges:        2,358,    9,504KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            10               Timer unplugs:           2
CPU2 (ext4.trace):
 Reads Queued:         191,      844KiB  Writes Queued:           2,        3KiB
 Read Dispatches:      191,      844KiB  Write Dispatches:        1,        3KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:      144,      684KiB  Writes Completed:       10,    2,608KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:            29               Write depth:            32
 IO unplugs:             6               Timer unplugs:           0
CPU3 (ext4.trace):
 Reads Queued:       1,273,    5,116KiB  Writes Queued:         852,  112,049KiB
 Read Dispatches:    1,315,    5,116KiB  Write Dispatches:       72,  109,545KiB
 Reads Requeued:        42               Writes Requeued:        11
 Reads Completed:    1,121,    4,496KiB  Writes Completed:       56,    4,024KiB
 Read Merges:            0,        0KiB  Write Merges:          779,    3,392KiB
 Read depth:            29               Write depth:            32
 IO unplugs:            15               Timer unplugs:          11

Total (ext4.trace):
 Reads Queued:       4,955,   20,388KiB  Writes Queued:       5,377,  540,404KiB
 Read Dispatches:    5,046,   20,388KiB  Write Dispatches:      448,  540,404KiB
 Reads Requeued:        91               Writes Requeued:       126
 Reads Completed:    4,955,   20,388KiB  Writes Completed:      330,   23,072KiB
 Read Merges:            0,        0KiB  Write Merges:        5,051,   20,552KiB
 IO unplugs:            43               Timer unplugs:          13

Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 -   0.0%)




Peter was right then.  Per multiple recommendations
I'm switching to fstrim, it is quicker than the deletes.

fstrim consistently takes about a minute to run on a 40GB
volume.

time fstrim -v /usr
/usr: 9047117824 bytes were trimmed

real    0m56.121s
user    0m0.090s
sys     0m0.000s

I see that the running of fstrim was already discussed
http://oss.sgi.com/archives/xfs/2011-05/msg00338.html.

Please add something to the FAQ about using SSDs.  It would be great if
people could see the recommended mount options and fstrim crontab entry
(or other option) for running xfs on a SSD.  

Thanks for the input and patience.  

~tom

Attachment: ext4.trace.blktrace.tar.bz
Description: application/bzip-compressed-tar

Attachment: signature.asc
Description: This is a digitally signed message part

<Prev in Thread] Current Thread [Next in Thread>