Poor performance using discard
Thomas Lynema
lyz27 at yahoo.com
Fri Mar 2 09:41:50 CST 2012
[....]
> Any errors in dmesg?
None
> Also, I think you need to provide a block trace (output of
> blktrace/blkparse for the rm -rf workloads) for both the XFS and
> ext4 cases
Here's the output from running similar
commands as Christoph:
blkparse xfs.trace.blktrace.* | grep -w D | wc -l
1134996
blkparse ext4.trace.blktrace.* | grep -w D | wc -l
27240
Attached archives of half of the ext4 results.
Summary:
CPU0 (ext4.trace):
Reads Queued: 372, 1,944KiB Writes Queued: 1,932, 7,728KiB
Read Dispatches: 372, 1,944KiB Write Dispatches: 19, 7,756KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 419, 2,104KiB Writes Completed: 13, 5,123KiB
Read Merges: 0, 0KiB Write Merges: 1,914, 7,656KiB
Read depth: 29 Write depth: 32
IO unplugs: 12 Timer unplugs: 0
CPU1 (ext4.trace):
Reads Queued: 3,119, 12,484KiB Writes Queued: 2,591, 420,624KiB
Read Dispatches: 3,168, 12,484KiB Write Dispatches: 356, 423,100KiB
Reads Requeued: 49 Writes Requeued: 115
Reads Completed: 3,271, 13,104KiB Writes Completed: 251, 11,317KiB
Read Merges: 0, 0KiB Write Merges: 2,358, 9,504KiB
Read depth: 29 Write depth: 32
IO unplugs: 10 Timer unplugs: 2
CPU2 (ext4.trace):
Reads Queued: 191, 844KiB Writes Queued: 2, 3KiB
Read Dispatches: 191, 844KiB Write Dispatches: 1, 3KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 144, 684KiB Writes Completed: 10, 2,608KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 29 Write depth: 32
IO unplugs: 6 Timer unplugs: 0
CPU3 (ext4.trace):
Reads Queued: 1,273, 5,116KiB Writes Queued: 852, 112,049KiB
Read Dispatches: 1,315, 5,116KiB Write Dispatches: 72, 109,545KiB
Reads Requeued: 42 Writes Requeued: 11
Reads Completed: 1,121, 4,496KiB Writes Completed: 56, 4,024KiB
Read Merges: 0, 0KiB Write Merges: 779, 3,392KiB
Read depth: 29 Write depth: 32
IO unplugs: 15 Timer unplugs: 11
Total (ext4.trace):
Reads Queued: 4,955, 20,388KiB Writes Queued: 5,377, 540,404KiB
Read Dispatches: 5,046, 20,388KiB Write Dispatches: 448, 540,404KiB
Reads Requeued: 91 Writes Requeued: 126
Reads Completed: 4,955, 20,388KiB Writes Completed: 330, 23,072KiB
Read Merges: 0, 0KiB Write Merges: 5,051, 20,552KiB
IO unplugs: 43 Timer unplugs: 13
Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 - 0.0%)
CPU0 (ext4.trace):
Reads Queued: 372, 1,944KiB Writes Queued: 1,932, 7,728KiB
Read Dispatches: 372, 1,944KiB Write Dispatches: 19, 7,756KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 419, 2,104KiB Writes Completed: 13, 5,123KiB
Read Merges: 0, 0KiB Write Merges: 1,914, 7,656KiB
Read depth: 29 Write depth: 32
IO unplugs: 12 Timer unplugs: 0
CPU1 (ext4.trace):
Reads Queued: 3,119, 12,484KiB Writes Queued: 2,591, 420,624KiB
Read Dispatches: 3,168, 12,484KiB Write Dispatches: 356, 423,100KiB
Reads Requeued: 49 Writes Requeued: 115
Reads Completed: 3,271, 13,104KiB Writes Completed: 251, 11,317KiB
Read Merges: 0, 0KiB Write Merges: 2,358, 9,504KiB
Read depth: 29 Write depth: 32
IO unplugs: 10 Timer unplugs: 2
CPU2 (ext4.trace):
Reads Queued: 191, 844KiB Writes Queued: 2, 3KiB
Read Dispatches: 191, 844KiB Write Dispatches: 1, 3KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 144, 684KiB Writes Completed: 10, 2,608KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 29 Write depth: 32
IO unplugs: 6 Timer unplugs: 0
CPU3 (ext4.trace):
Reads Queued: 1,273, 5,116KiB Writes Queued: 852, 112,049KiB
Read Dispatches: 1,315, 5,116KiB Write Dispatches: 72, 109,545KiB
Reads Requeued: 42 Writes Requeued: 11
Reads Completed: 1,121, 4,496KiB Writes Completed: 56, 4,024KiB
Read Merges: 0, 0KiB Write Merges: 779, 3,392KiB
Read depth: 29 Write depth: 32
IO unplugs: 15 Timer unplugs: 11
Total (ext4.trace):
Reads Queued: 4,955, 20,388KiB Writes Queued: 5,377, 540,404KiB
Read Dispatches: 5,046, 20,388KiB Write Dispatches: 448, 540,404KiB
Reads Requeued: 91 Writes Requeued: 126
Reads Completed: 4,955, 20,388KiB Writes Completed: 330, 23,072KiB
Read Merges: 0, 0KiB Write Merges: 5,051, 20,552KiB
IO unplugs: 43 Timer unplugs: 13
Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 - 0.0%)
CPU0 (ext4.trace):
Reads Queued: 372, 1,944KiB Writes Queued: 1,932, 7,728KiB
Read Dispatches: 372, 1,944KiB Write Dispatches: 19, 7,756KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 419, 2,104KiB Writes Completed: 13, 5,123KiB
Read Merges: 0, 0KiB Write Merges: 1,914, 7,656KiB
Read depth: 29 Write depth: 32
IO unplugs: 12 Timer unplugs: 0
CPU1 (ext4.trace):
Reads Queued: 3,119, 12,484KiB Writes Queued: 2,591, 420,624KiB
Read Dispatches: 3,168, 12,484KiB Write Dispatches: 356, 423,100KiB
Reads Requeued: 49 Writes Requeued: 115
Reads Completed: 3,271, 13,104KiB Writes Completed: 251, 11,317KiB
Read Merges: 0, 0KiB Write Merges: 2,358, 9,504KiB
Read depth: 29 Write depth: 32
IO unplugs: 10 Timer unplugs: 2
CPU2 (ext4.trace):
Reads Queued: 191, 844KiB Writes Queued: 2, 3KiB
Read Dispatches: 191, 844KiB Write Dispatches: 1, 3KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 144, 684KiB Writes Completed: 10, 2,608KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 29 Write depth: 32
IO unplugs: 6 Timer unplugs: 0
CPU3 (ext4.trace):
Reads Queued: 1,273, 5,116KiB Writes Queued: 852, 112,049KiB
Read Dispatches: 1,315, 5,116KiB Write Dispatches: 72, 109,545KiB
Reads Requeued: 42 Writes Requeued: 11
Reads Completed: 1,121, 4,496KiB Writes Completed: 56, 4,024KiB
Read Merges: 0, 0KiB Write Merges: 779, 3,392KiB
Read depth: 29 Write depth: 32
IO unplugs: 15 Timer unplugs: 11
Total (ext4.trace):
Reads Queued: 4,955, 20,388KiB Writes Queued: 5,377, 540,404KiB
Read Dispatches: 5,046, 20,388KiB Write Dispatches: 448, 540,404KiB
Reads Requeued: 91 Writes Requeued: 126
Reads Completed: 4,955, 20,388KiB Writes Completed: 330, 23,072KiB
Read Merges: 0, 0KiB Write Merges: 5,051, 20,552KiB
IO unplugs: 43 Timer unplugs: 13
Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 - 0.0%)
CPU0 (ext4.trace):
Reads Queued: 372, 1,944KiB Writes Queued: 1,932, 7,728KiB
Read Dispatches: 372, 1,944KiB Write Dispatches: 19, 7,756KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 419, 2,104KiB Writes Completed: 13, 5,123KiB
Read Merges: 0, 0KiB Write Merges: 1,914, 7,656KiB
Read depth: 29 Write depth: 32
IO unplugs: 12 Timer unplugs: 0
CPU1 (ext4.trace):
Reads Queued: 3,119, 12,484KiB Writes Queued: 2,591, 420,624KiB
Read Dispatches: 3,168, 12,484KiB Write Dispatches: 356, 423,100KiB
Reads Requeued: 49 Writes Requeued: 115
Reads Completed: 3,271, 13,104KiB Writes Completed: 251, 11,317KiB
Read Merges: 0, 0KiB Write Merges: 2,358, 9,504KiB
Read depth: 29 Write depth: 32
IO unplugs: 10 Timer unplugs: 2
CPU2 (ext4.trace):
Reads Queued: 191, 844KiB Writes Queued: 2, 3KiB
Read Dispatches: 191, 844KiB Write Dispatches: 1, 3KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 144, 684KiB Writes Completed: 10, 2,608KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 29 Write depth: 32
IO unplugs: 6 Timer unplugs: 0
CPU3 (ext4.trace):
Reads Queued: 1,273, 5,116KiB Writes Queued: 852, 112,049KiB
Read Dispatches: 1,315, 5,116KiB Write Dispatches: 72, 109,545KiB
Reads Requeued: 42 Writes Requeued: 11
Reads Completed: 1,121, 4,496KiB Writes Completed: 56, 4,024KiB
Read Merges: 0, 0KiB Write Merges: 779, 3,392KiB
Read depth: 29 Write depth: 32
IO unplugs: 15 Timer unplugs: 11
Total (ext4.trace):
Reads Queued: 4,955, 20,388KiB Writes Queued: 5,377, 540,404KiB
Read Dispatches: 5,046, 20,388KiB Write Dispatches: 448, 540,404KiB
Reads Requeued: 91 Writes Requeued: 126
Reads Completed: 4,955, 20,388KiB Writes Completed: 330, 23,072KiB
Read Merges: 0, 0KiB Write Merges: 5,051, 20,552KiB
IO unplugs: 43 Timer unplugs: 13
Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 - 0.0%)
CPU0 (ext4.trace):
Reads Queued: 372, 1,944KiB Writes Queued: 1,932, 7,728KiB
Read Dispatches: 372, 1,944KiB Write Dispatches: 19, 7,756KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 419, 2,104KiB Writes Completed: 13, 5,123KiB
Read Merges: 0, 0KiB Write Merges: 1,914, 7,656KiB
Read depth: 29 Write depth: 32
IO unplugs: 12 Timer unplugs: 0
CPU1 (ext4.trace):
Reads Queued: 3,119, 12,484KiB Writes Queued: 2,591, 420,624KiB
Read Dispatches: 3,168, 12,484KiB Write Dispatches: 356, 423,100KiB
Reads Requeued: 49 Writes Requeued: 115
Reads Completed: 3,271, 13,104KiB Writes Completed: 251, 11,317KiB
Read Merges: 0, 0KiB Write Merges: 2,358, 9,504KiB
Read depth: 29 Write depth: 32
IO unplugs: 10 Timer unplugs: 2
CPU2 (ext4.trace):
Reads Queued: 191, 844KiB Writes Queued: 2, 3KiB
Read Dispatches: 191, 844KiB Write Dispatches: 1, 3KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 144, 684KiB Writes Completed: 10, 2,608KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 29 Write depth: 32
IO unplugs: 6 Timer unplugs: 0
CPU3 (ext4.trace):
Reads Queued: 1,273, 5,116KiB Writes Queued: 852, 112,049KiB
Read Dispatches: 1,315, 5,116KiB Write Dispatches: 72, 109,545KiB
Reads Requeued: 42 Writes Requeued: 11
Reads Completed: 1,121, 4,496KiB Writes Completed: 56, 4,024KiB
Read Merges: 0, 0KiB Write Merges: 779, 3,392KiB
Read depth: 29 Write depth: 32
IO unplugs: 15 Timer unplugs: 11
Total (ext4.trace):
Reads Queued: 4,955, 20,388KiB Writes Queued: 5,377, 540,404KiB
Read Dispatches: 5,046, 20,388KiB Write Dispatches: 448, 540,404KiB
Reads Requeued: 91 Writes Requeued: 126
Reads Completed: 4,955, 20,388KiB Writes Completed: 330, 23,072KiB
Read Merges: 0, 0KiB Write Merges: 5,051, 20,552KiB
IO unplugs: 43 Timer unplugs: 13
Throughput (R/W): 5,011KiB/s / 5,671KiB/s
Events (ext4.trace): 57,930 entries
Skips: 0 forward (0 - 0.0%)
Peter was right then. Per multiple recommendations
I'm switching to fstrim, it is quicker than the deletes.
fstrim consistently takes about a minute to run on a 40GB
volume.
time fstrim -v /usr
/usr: 9047117824 bytes were trimmed
real 0m56.121s
user 0m0.090s
sys 0m0.000s
I see that the running of fstrim was already discussed
http://oss.sgi.com/archives/xfs/2011-05/msg00338.html.
Please add something to the FAQ about using SSDs. It would be great if
people could see the recommended mount options and fstrim crontab entry
(or other option) for running xfs on a SSD.
Thanks for the input and patience.
~tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ext4.trace.blktrace.tar.bz
Type: application/x-bzip-compressed-tar
Size: 364717 bytes
Desc: not available
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20120302/83020ab0/attachment-0001.tbz2>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20120302/83020ab0/attachment-0001.sig>
More information about the xfs
mailing list