Search String: Display: Description: Sort:

Results:

References: [ +subject:/^(?:^\s*(re|sv|fwd|fw)[\[\]\d]*[:>-]+\s*)*deleting\s+2TB\s+lots\s+of\s+files\s+with\s+delaylog\:\s+sync\s+helps\?\s*$/: 17 ]

Total 17 documents matching your query.

1. deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 01:30:41 +0200
I'm just trying the delaylog mount option on a filesystem (LVM over 2x 2TB 4K sector drives), and I see this while running 8 processes of "rm -r * & 2>/dev/null": Device: rrqm/s wrqm/s r/s w/s rkB/s
/archives/xfs/2010-08/msg00337.html (10,244 bytes)

2. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 10:06:31 +1000
You're probably getting RMW cycles on inode writeback. I've been noticing this lately with my benchmarking - the VM is being _very aggressive_ reclaiming page cache pages vs inode caches and as a res
/archives/xfs/2010-08/msg00340.html (11,268 bytes)

3. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 02:22:31 +0200
Nice explanation. This is a hexa-core AMD Phenom(tm) II X6 1090T Processor with up to 3.2GHz per core, so that shouldn't be - or is there only one core used? I think I read somewhere that each AG sho
/archives/xfs/2010-08/msg00341.html (10,851 bytes)

4. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Tue, 31 Aug 2010 22:01:47 -0500
Dave Chinner put forth on 8/31/2010 7:06 PM: 7200 rpm is the highest spindle speed for 2TB drives--5400 is most common. None of them are going to do much over 200 random seeks/second, if that. That's
/archives/xfs/2010-08/msg00344.html (8,428 bytes)

5. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 13:19:54 +1000
I'm getting a 8core/16thread server being CPU bound with multithreaded unlink workloads using delaylog, so it's entirely possible that all CPU cores are fully utilised on your machine. If all the fil
/archives/xfs/2010-08/msg00345.html (10,399 bytes)

6. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 13:41:56 +1000
Absolutely. Nothing in XFS is simple. ;) Unlinks that free the inode clusters results in no inode writeback load, so the majority of the IO is log traffic. Hence they are either log IO bound or read
/archives/xfs/2010-08/msg00346.html (9,653 bytes)

7. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Tue, 31 Aug 2010 23:42:07 -0500
Dave Chinner put forth on 8/31/2010 10:19 PM: What's your disk configuration on this 8 core machine? Are you implying/stating that the performance of the disk subsystem is irrelevant WRT multithreade
/archives/xfs/2010-08/msg00347.html (9,174 bytes)

8. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 16:44:39 +1000
Depends on where I place the disk image for the VM's I run on it ;) For example, running fs_mark with 4 threads to create then delete 200k files in a directory per thread in a 4p VM w/ 2GB RAM with t
/archives/xfs/2010-09/msg00002.html (12,133 bytes)

9. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx>
Date: Wed, 1 Sep 2010 09:45:58 +0200
Just as Stan I'm puzzled by this. Why is it such a hard work for the CPU, what does it do? Is it really about calculating something, or has it to do with lock contention, cold caches, cache line bou
/archives/xfs/2010-09/msg00004.html (9,071 bytes)

10. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Thu, 2 Sep 2010 11:17:44 +1000
Ok, it seems that people don't have any real idea of th complexity of directory operations in XFS, so I'll give you a quick overview. The XFS directory structure is excedingly complex and the algorit
/archives/xfs/2010-09/msg00017.html (12,213 bytes)

11. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx>
Date: Thu, 2 Sep 2010 04:15:47 +0200
Is the SSD-needed "trim" belonging into here? Now I understand, thanks again for that great explanation. Time to invent the "XFS rm co-processor". Should be multi-core so it scales better. Maybe some
/archives/xfs/2010-09/msg00019.html (10,199 bytes)

12. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Thu, 02 Sep 2010 00:37:39 -0500
Dave Chinner put forth on 9/1/2010 1:44 AM: Is this a single socket quad core Intel machine with hyperthreading enabled? That would fully explain the results above. Looks like you ran out of memory b
/archives/xfs/2010-09/msg00022.html (12,575 bytes)

13. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Thu, 2 Sep 2010 17:01:08 +1000
No, It's a dual socket (8c/16t) server. No, that's definitely not the case. A different kernel in the same 8p VM, 12x2TB SAS storage, w/ 4 threads, mount options "logbsize=262144" FSUse% Count Size F
/archives/xfs/2010-09/msg00023.html (12,612 bytes)

14. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Thu, 02 Sep 2010 02:51:21 -0500
Dave Chinner put forth on 9/1/2010 8:17 PM: More like an FPGA. As we see on list, daily, the XFS code changes far too rapidly for implementation in an ASIC. ;) Hay, there's a sales opportunity for SG
/archives/xfs/2010-09/msg00025.html (8,938 bytes)

15. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Thu, 02 Sep 2010 03:41:59 -0500
Dave Chinner put forth on 9/2/2010 2:01 AM: What happens when you bump each of these to 8 threads, 1 per core? If the test consumes all cpus/cores, what instrumentation are you viewing that tells you
/archives/xfs/2010-09/msg00026.html (11,828 bytes)

16. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Thu, 2 Sep 2010 21:29:20 +1000
FSUse% Count Size Files/sec App Overhead 0 1600000 0 127979.3 13156823 So, 1 thread does 19k files/s, 2 thread does 37k files/s, 4 gets 67k, and 8 gets 128k. I'd say that's almost linear scaling and
/archives/xfs/2010-09/msg00029.html (16,050 bytes)

17. Re: deleting 2TB lots of files with delaylog: sync helps? (score: 1)
Author: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Thu, 02 Sep 2010 09:57:43 -0500
Thanks Dave. I don't normally top post, but I just wanted to quickly say I _really_ enjoyed reading your reply below. It was seriously educational. I really enjoyed your note about the 24p Altix syst
/archives/xfs/2010-09/msg00031.html (17,868 bytes)


This search system is powered by Namazu