From owner-xfs@oss.sgi.com Sun Apr 1 15:45:13 2007 Received: with ECARTIS (v1.0.0; list xfs); Sun, 01 Apr 2007 15:45:16 -0700 (PDT) X-Spam-oss-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.0-pre1-r499012 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l31MjA6p010169 for ; Sun, 1 Apr 2007 15:45:11 -0700 Received: from snort.melbourne.sgi.com (snort.melbourne.sgi.com [134.14.54.149]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id IAA12073; Mon, 2 Apr 2007 08:45:01 +1000 Received: from snort.melbourne.sgi.com (localhost [127.0.0.1]) by snort.melbourne.sgi.com (SGI-8.12.5/8.12.5) with ESMTP id l31Mj0Af40434916; Mon, 2 Apr 2007 08:45:00 +1000 (AEST) Received: (from dgc@localhost) by snort.melbourne.sgi.com (SGI-8.12.5/8.12.5/Submit) id l31MiwTX46334670; Mon, 2 Apr 2007 08:44:58 +1000 (AEST) Date: Mon, 2 Apr 2007 08:44:58 +1000 From: David Chinner To: Martin Steigerwald Cc: linux-xfs@oss.sgi.com Subject: Re: write barrier and USB devices Message-ID: <20070401224458.GR32597093@melbourne.sgi.com> References: <200703301523.58027.Martin@lichtvoll.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200703301523.58027.Martin@lichtvoll.de> User-Agent: Mutt/1.4.2.1i X-archive-position: 11015 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: dgc@sgi.com Precedence: bulk X-list: xfs On Fri, Mar 30, 2007 at 03:23:57PM +0200, Martin Steigerwald wrote: > > Hello! > > Does the usb mass storage driver support write barriers? You should ask the usb folks that question.... Cheers, Dave. -- Dave Chinner Principal Engineer SGI Australian Software Group From owner-xfs@oss.sgi.com Mon Apr 2 11:24:06 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 11:24:12 -0700 (PDT) X-Spam-oss-Status: No, score=-0.6 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l32IO2kH008050 for ; Mon, 2 Apr 2007 11:24:04 -0700 Received: from boing.melbourne.sgi.com (boing.melbourne.sgi.com [134.14.55.141]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id QAA23891; Mon, 2 Apr 2007 16:30:51 +1000 Date: Mon, 02 Apr 2007 16:31:52 +1100 From: Timothy Shimmin To: xfs-dev@sgi.com, xfs@oss.sgi.com Subject: review: remove unused ilen var from xfs_vnodeops.c Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 11016 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tes@sgi.com Precedence: bulk X-list: xfs simple cleanup patch =========================================================================== Index: fs/xfs/xfs_vnodeops.c =========================================================================== --- a/fs/xfs/xfs_vnodeops.c 2007-04-02 15:56:30.000000000 +1000 +++ b/fs/xfs/xfs_vnodeops.c 2007-04-02 15:19:42.926081759 +1000 @@ -4289,7 +4289,6 @@ xfs_free_file_space( int error; xfs_fsblock_t firstfsb; xfs_bmap_free_t free_list; - xfs_off_t ilen; xfs_bmbt_irec_t imap; xfs_off_t ioffset; xfs_extlen_t mod=0; @@ -4338,10 +4337,7 @@ xfs_free_file_space( } rounding = max_t(uint, 1 << mp->m_sb.sb_blocklog, NBPP); - ilen = len + (offset & (rounding - 1)); ioffset = offset & ~(rounding - 1); - if (ilen & (rounding - 1)) - ilen = (ilen + rounding) & ~(rounding - 1); if (VN_CACHED(vp) != 0) { xfs_inval_cached_trace(&ip->i_iocore, ioffset, -1, ============================================================================= I think ilen was removed a while back when we changed a call interface... revision 1.534 date: 2002/07/08 22:09:30; author: lord; state: Exp; lines: +1 -2 modid: 2.4.x-xfs:slinx:122666a changes xfs_inval_cached_pages interface =========================================================================== Index: fs/xfs/xfs_vnodeops.c =========================================================================== --- a/fs/xfs/xfs_vnodeops.c 2007-04-02 15:38:34.000000000 +1000 +++ b/fs/xfs/xfs_vnodeops.c 2007-04-02 15:38:34.000000000 +1000 @@ -5459,8 +5459,7 @@ xfs_free_file_space( ioffset = offset & ~(rounding - 1); if (ilen & (rounding - 1)) ilen = (ilen + rounding) & ~(rounding - 1); - xfs_inval_cached_pages(XFS_ITOV(ip), &(ip->i_iocore), - ioffset, ilen, NULL, 0); + xfs_inval_cached_pages(XFS_ITOV(ip), &(ip->i_iocore), ioffset, 0, 0); /* * Need to zero the stuff we're not freeing, on disk. * If its specrt (realtime & can't use unwritten extents) then --Tim From owner-xfs@oss.sgi.com Mon Apr 2 11:35:11 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 11:35:14 -0700 (PDT) X-Spam-oss-Status: No, score=0.0 required=5.0 tests=BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail.g-house.de (ns2.g-housing.de [81.169.133.75]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l32IXskH011758 for ; Mon, 2 Apr 2007 11:35:11 -0700 Received: from [77.99.119.196] (helo=77-99-119-196.cable.ubr04.linl.blueyonder.co.uk) by mail.g-house.de with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1HYR6a-0002wK-26 for xfs@oss.sgi.com; Mon, 02 Apr 2007 20:18:12 +0200 Date: Mon, 2 Apr 2007 19:18:01 +0100 (BST) From: Christian Kujau X-X-Sender: evil@sheep.housecafe.de To: xfs@oss.sgi.com Subject: possible recursive locking detected Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-archive-position: 11017 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: lists@nerdbynature.de Precedence: bulk X-list: xfs Hi, when I enabled a few more debug-options in the kernel (vanilla 2.6.21-rc5), I came across: [ INFO: possible recursive locking detected ] 2.6.21-rc5 #2 --------------------------------------------- rm/32198 is trying to acquire lock: xfs_ilock+0x71/0xa0 but task is already holding lock: xfs_ilock+0x71/0xa0 other info that might help us debug this: 3 locks held by rm/32198: do_unlinkat+0x96/0x160 vfs_unlink+0x75/0xe0 xfs_ilock+0x71/0xa0 stack backtrace: __lock_acquire+0xa99/0x1010 lock_acquire+0x57/0x70 xfs_ilock+0x71/0xa0 down_write+0x38/0x50 xfs_ilock+0x71/0xa0 xfs_ilock+0x71/0xa0 xfs_lock_dir_and_entry+0xf6/0x100 xfs_remove+0x197/0x4e0 d_instantiate+0x19/0x40 d_rehash+0x20/0x50 vfs_unlink+0x75/0xe0 xfs_vn_unlink+0x23/0x60 __mutex_lock_slowpath+0x13f/0x280 mark_held_locks+0x6b/0x90 __mutex_lock_slowpath+0x13f/0x280 __mutex_lock_slowpath+0x13f/0x280 trace_hardirqs_on+0xb9/0x160 vfs_unlink+0x75/0xe0 __mutex_lock_slowpath+0x132/0x280 vfs_unlink+0x75/0xe0 permission+0x91/0xf0 vfs_unlink+0x89/0xe0 do_unlinkat+0xd2/0x160 sysenter_past_esp+0x8d/0x99 trace_hardirqs_on+0xb9/0x160 sysenter_past_esp+0x5d/0x99 ======================= Is this something I have to worry about? Please see http://nerdbynature.de/bits/2.6.21-rc5/ for a few more details. Thanks, Christian. -- BOFH excuse #372: Forced to support NT servers; sysadmins quit. From owner-xfs@oss.sgi.com Mon Apr 2 13:19:25 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 13:19:27 -0700 (PDT) X-Spam-oss-Status: No, score=-1.3 required=5.0 tests=AWL,BAYES_20 autolearn=no version=3.2.0-pre1-r499012 Received: from pentafluge.infradead.org (pentafluge.infradead.org [213.146.154.40]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l32KJOfB015559 for ; Mon, 2 Apr 2007 13:19:25 -0700 Received: from hch by pentafluge.infradead.org with local (Exim 4.63 #1 (Red Hat Linux)) id 1HYSjB-0006XR-RE; Mon, 02 Apr 2007 21:02:09 +0100 Date: Mon, 2 Apr 2007 21:02:09 +0100 From: Christoph Hellwig To: Timothy Shimmin Cc: xfs-dev@sgi.com, xfs@oss.sgi.com Subject: Re: review: remove unused ilen var from xfs_vnodeops.c Message-ID: <20070402200209.GA25101@infradead.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-SRS-Rewrite: SMTP reverse-path rewritten from by pentafluge.infradead.org See http://www.infradead.org/rpr.html X-archive-position: 11018 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: hch@infradead.org Precedence: bulk X-list: xfs On Mon, Apr 02, 2007 at 04:31:52PM +1100, Timothy Shimmin wrote: > simple cleanup patch looks good. From owner-xfs@oss.sgi.com Mon Apr 2 14:44:47 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 14:44:50 -0700 (PDT) X-Spam-oss-Status: No, score=3.0 required=5.0 tests=BAYES_95,HTML_MESSAGE autolearn=no version=3.2.0-pre1-r499012 Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.243]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l32LijfB006582 for ; Mon, 2 Apr 2007 14:44:46 -0700 Received: by an-out-0708.google.com with SMTP id c5so1347883anc for ; Mon, 02 Apr 2007 14:44:44 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=B0dKOXK/Z19QmQgxQu99WBEx++uR/fqbhpRomvr1bdl0Go0keJ5m4MEuIy4+8IbZbuj/tKmtmSIhGlVq68O7zFhNJj/YYQTVX3Ww3e5egV2T9HRYdwH4k3r0Stdep02hmpeRp05IoOK5iLST4BmJFwXeo8jk8ESupxftDNe7WiU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=Pj9/4vi7bihLMXtvW2pCj7KVnv8jPvl2JSdnm0yBUnbZ1A0Se9Jqx7YsGqs6OIy0rbDBhPLrQ6AqHeOYCivaz7VLdgbi1g4kl+Jh0oRS0bBWJURBb74AgSkaLQs6teRmu/MAbrndaOH0MWOPIZHjLgkSq6n/YX0IjgltU5V8zO8= Received: by 10.100.144.11 with SMTP id r11mr3858808and.1175548737613; Mon, 02 Apr 2007 14:18:57 -0700 (PDT) Received: by 10.100.200.11 with HTTP; Mon, 2 Apr 2007 14:18:57 -0700 (PDT) Message-ID: <817da7960704021418g6e1d4662y3250be14bc01ab69@mail.gmail.com> Date: Mon, 2 Apr 2007 17:18:57 -0400 From: "Charles Weber" To: "Roger Heflin" Subject: Re: xfs partial dismount issue Cc: linux-xfs@oss.sgi.com, sandeen@sandeen.net In-Reply-To: <45EC868A.4060607@atipa.com> MIME-Version: 1.0 References: <45EC3DEA.3000105@sandeen.net> <45EC868A.4060607@atipa.com> Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 7bit Content-length: 2822 X-archive-position: 11019 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: chaweber@gmail.com Precedence: bulk X-list: xfs Well actually I did a test with ext3 and got the same result. It partially dismounted after a day or so of use and seemed identical to my previous xfs filesystem failures . My guess now is that this has always occurred when all 6 raid controllers (2 per card) were in use. I could go quite some time with 5 of the 6 controllers used. I consolidated everything to 2 cards, removed one card and put in fiber channel card for my new storage array So far no problems. If so then it seems something is funny about the cciss driver. thanks, Chuck On 3/5/07, Roger Heflin wrote: > > Charles Weber wrote: > > Eric Sandeen sandeen.net> writes: > > > >> Chuck Weber wrote: > >>> Hi everyone, I have a long running problem perhaps you can help with. > I > >>> will include as much detail as I can. I can set up a spare server-disk > >>> set for testing if you have any bright ideas. > >>> > >>> We use XFS for samba and nfs on x86_64 Fedora Proliant DL585/385 > >>> servers. Our busiest server has disk partitions go away. > >> What do you mean by this, exactly? The partitions themselves go away, > >> or are you talking about the problem described below where processes > >> start hanging? > >> > > Here is an example partition (1 of 6 or more xfs storage only). > > /share/store3 with samba shares on /share/store3/lls, lds, lxs and so > on. > > I will get a call saying my groups share (lxs) is no longer accessable. > I ssh > > into server and can ls /share/store3 but ls will hang when I ls > > /share/store3/lxs. Shortly there after ls will hang for the root or any > > directory on the partition. Other partitions will be fine and other > samba shares > > will be fine until the queued up process load bogs the server down. > > > > Charles, > > I have seen what may be a similar issue on SLES9SP2, we had 1 xfs > partition, and under certain conditions it would stop responding, all > non-xfs partitions were ok, and everything was fine after a reboot. > > Under sysrq-t it appeared to me that 2 separate processes were calling > fsync and were causing each other to deadlock (and locking all others > out of changing the xfs partition). I was not able to determine exactly > what the underlying bug was, but all of the hung processes > were waiting on locks in at least several widely different parts of the > xfs and kernel code, and adjusting the application to not fsync has > apparently resulted in the deadlock not occuring. In this case > there were multiple (2-4) different instances of the application calling > fsync apparently sometimes at close to the same time. With the > given application the failure was almost a certainly on one machine > (of 100) running the application overnight. > > Roger > [[HTML alternate version deleted]] From owner-xfs@oss.sgi.com Mon Apr 2 16:46:21 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 16:46:22 -0700 (PDT) X-Spam-oss-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00 autolearn=no version=3.2.0-pre1-r499012 Received: from mx1.redhat.com (mx1.redhat.com [66.187.233.31]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l32NkIfB010080 for ; Mon, 2 Apr 2007 16:46:20 -0700 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l32NkHUX007234; Mon, 2 Apr 2007 19:46:17 -0400 Received: from pobox-2.corp.redhat.com (pobox-2.corp.redhat.com [10.11.255.15]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l32NkHod021971; Mon, 2 Apr 2007 19:46:17 -0400 Received: from [10.15.80.10] (neon.msp.redhat.com [10.15.80.10]) by pobox-2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l32NkGlR000765; Mon, 2 Apr 2007 19:46:16 -0400 Message-ID: <46119596.1020900@sandeen.net> Date: Mon, 02 Apr 2007 18:45:26 -0500 From: Eric Sandeen User-Agent: Thunderbird 1.5.0.10 (X11/20070302) MIME-Version: 1.0 To: Timothy Shimmin CC: xfs-dev@sgi.com, xfs@oss.sgi.com Subject: Re: review: remove unused ilen var from xfs_vnodeops.c References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-archive-position: 11020 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: sandeen@sandeen.net Precedence: bulk X-list: xfs Timothy Shimmin wrote: > simple cleanup patch > Hey that's my job! ;-) Looks good -Eric From owner-xfs@oss.sgi.com Mon Apr 2 17:54:12 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 17:54:15 -0700 (PDT) X-Spam-oss-Status: No, score=-1.9 required=5.0 tests=AWL,BAYES_00 autolearn=no version=3.2.0-pre1-r499012 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l330s9fB000643 for ; Mon, 2 Apr 2007 17:54:11 -0700 Received: from chook.melbourne.sgi.com (chook.melbourne.sgi.com [134.14.54.237]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id KAA23394 for ; Tue, 3 Apr 2007 10:54:07 +1000 Received: by chook.melbourne.sgi.com (Postfix, from userid 1116) id C15D458FF80A; Tue, 3 Apr 2007 10:54:07 +1000 (EST) To: xfs@oss.sgi.com Subject: TAKE cleanup - remove ilen refs from vnodeops.c Message-Id: <20070403005407.C15D458FF80A@chook.melbourne.sgi.com> Date: Tue, 3 Apr 2007 10:54:07 +1000 (EST) From: tes@sgi.com (Tim Shimmin) X-archive-position: 11021 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tes@sgi.com Precedence: bulk X-list: xfs Remove unused ilen variable and references. Date: Tue Apr 3 10:53:20 AEST 2007 Workarea: chook.melbourne.sgi.com:/build/tes/2.6.x-xfs Inspected by: lachlan@sgi.com,sandeen@sandeen.net The following file(s) were checked into: longdrop.melbourne.sgi.com:/isms/linux/2.6.x-xfs-melb Modid: xfs-linux-melb:xfs-kern:28344a fs/xfs/xfs_vnodeops.c - 1.694 - changed http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-linux/xfs_vnodeops.c.diff?r1=text&tr1=1.694&r2=text&tr2=1.693&f=h - Remove unused ilen variable and references. From owner-xfs@oss.sgi.com Mon Apr 2 23:09:38 2007 Received: with ECARTIS (v1.0.0; list xfs); Mon, 02 Apr 2007 23:09:40 -0700 (PDT) X-Spam-oss-Status: No, score=-0.6 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l3369YfB022611 for ; Mon, 2 Apr 2007 23:09:36 -0700 Received: from boing.melbourne.sgi.com (boing.melbourne.sgi.com [134.14.55.141]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id QAA01072; Tue, 3 Apr 2007 16:09:33 +1000 Date: Tue, 03 Apr 2007 16:10:38 +1100 From: Timothy Shimmin To: xfs@oss.sgi.com, xfs-dev@sgi.com Subject: review: export xfs_buftarg_list for use by xfsidbg Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 11023 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tes@sgi.com Precedence: bulk X-list: xfs Content-Length: 6296 Lines: 185 Hi, This patch addresses the problem of having xfs_buftarg_list global for use by kdb and xfsidbg, where otherwise it would be static. If we are using xfsidbg, then we export the xfs_get_buftarg_list function to it. Previous to Dave's (dgc) static changes, we were only globalizing it if we were in DEBUG - when really it is a question of xfsidbg. Using CONFIG_KDB_MODULES as this is what we use in the Makefiles for determining if xfsidbg is used. --Tim linux-2.4/xfs_buf.c | 8 ++++++++ linux-2.4/xfs_buf.h | 3 +++ linux-2.4/xfs_ksyms.c | 5 ++--- linux-2.6/xfs_buf.c | 10 +++++++++- linux-2.6/xfs_buf.h | 3 +++ linux-2.6/xfs_ksyms.c | 5 ++--- xfsidbg.c | 9 +++------ 7 files changed, 30 insertions(+), 13 deletions(-) =========================================================================== Index: fs/xfs/linux-2.4/xfs_buf.c =========================================================================== --- a/fs/xfs/linux-2.4/xfs_buf.c 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.4/xfs_buf.c 2007-04-03 15:45:23.930213823 +1000 @@ -2335,3 +2335,11 @@ xfs_buf_terminate(void) kmem_zone_destroy(xfs_buf_zone); kmem_shake_deregister(xfs_buf_shake); } + +#ifdef CONFIG_KDB_MODULES +struct list_head * +xfs_get_buftarg_list(void) +{ + return &xfs_buftarg_list; +} +#endif =========================================================================== Index: fs/xfs/linux-2.4/xfs_buf.h =========================================================================== --- a/fs/xfs/linux-2.4/xfs_buf.h 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.4/xfs_buf.h 2007-04-03 15:46:59.997634360 +1000 @@ -500,6 +500,9 @@ extern void xfs_free_buftarg(xfs_buftarg extern void xfs_wait_buftarg(xfs_buftarg_t *); extern int xfs_setsize_buftarg(xfs_buftarg_t *, unsigned int, unsigned int); extern int xfs_flush_buftarg(xfs_buftarg_t *, int); +#ifdef CONFIG_KDB_MODULES +extern struct list_head *xfs_get_buftarg_list(void); +#endif #define xfs_getsize_buftarg(buftarg) block_size((buftarg)->bt_kdev) #define xfs_readonly_buftarg(buftarg) is_read_only((buftarg)->bt_kdev) =========================================================================== Index: fs/xfs/linux-2.4/xfs_ksyms.c =========================================================================== --- a/fs/xfs/linux-2.4/xfs_ksyms.c 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.4/xfs_ksyms.c 2007-04-03 15:46:29.489629322 +1000 @@ -124,9 +124,8 @@ EXPORT_SYMBOL(xfs_params); EXPORT_SYMBOL(xfs_bmbt_disk_get_all); #endif -#if defined(CONFIG_XFS_DEBUG) -extern struct list_head xfs_buftarg_list; -EXPORT_SYMBOL(xfs_buftarg_list); +#if defined(CONFIG_KDB_MODULES) +EXPORT_SYMBOL(xfs_get_buftarg_list); #endif /* =========================================================================== Index: fs/xfs/linux-2.6/xfs_buf.c =========================================================================== --- a/fs/xfs/linux-2.6/xfs_buf.c 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.6/xfs_buf.c 2007-04-03 15:11:15.778300965 +1000 @@ -1426,7 +1426,7 @@ xfs_free_bufhash( /* * buftarg list for delwrite queue processing */ -LIST_HEAD(xfs_buftarg_list); +static LIST_HEAD(xfs_buftarg_list); static DEFINE_SPINLOCK(xfs_buftarg_lock); STATIC void @@ -1867,3 +1867,11 @@ xfs_buf_terminate(void) ktrace_free(xfs_buf_trace_buf); #endif } + +#ifdef CONFIG_KDB_MODULES +struct list_head * +xfs_get_buftarg_list(void) +{ + return &xfs_buftarg_list; +} +#endif =========================================================================== Index: fs/xfs/linux-2.6/xfs_buf.h =========================================================================== --- a/fs/xfs/linux-2.6/xfs_buf.h 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.6/xfs_buf.h 2007-04-03 15:22:51.547106965 +1000 @@ -411,6 +411,9 @@ extern void xfs_free_buftarg(xfs_buftarg extern void xfs_wait_buftarg(xfs_buftarg_t *); extern int xfs_setsize_buftarg(xfs_buftarg_t *, unsigned int, unsigned int); extern int xfs_flush_buftarg(xfs_buftarg_t *, int); +#ifdef CONFIG_KDB_MODULES +extern struct list_head *xfs_get_buftarg_list(void); +#endif #define xfs_getsize_buftarg(buftarg) block_size((buftarg)->bt_bdev) #define xfs_readonly_buftarg(buftarg) bdev_read_only((buftarg)->bt_bdev) =========================================================================== Index: fs/xfs/linux-2.6/xfs_ksyms.c =========================================================================== --- a/fs/xfs/linux-2.6/xfs_ksyms.c 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/linux-2.6/xfs_ksyms.c 2007-04-03 14:52:27.011000730 +1000 @@ -125,9 +125,8 @@ EXPORT_SYMBOL(xfs_params); EXPORT_SYMBOL(xfs_bmbt_disk_get_all); #endif -#if defined(CONFIG_XFS_DEBUG) -extern struct list_head xfs_buftarg_list; -EXPORT_SYMBOL(xfs_buftarg_list); +#if defined(CONFIG_KDB_MODULES) +EXPORT_SYMBOL(xfs_get_buftarg_list); #endif /* =========================================================================== Index: fs/xfs/xfsidbg.c =========================================================================== --- a/fs/xfs/xfsidbg.c 2007-04-03 15:47:15.000000000 +1000 +++ b/fs/xfs/xfsidbg.c 2007-04-03 15:24:02.201877199 +1000 @@ -62,6 +62,7 @@ #include "xfs_quota.h" #include "quota/xfs_qm.h" #include "xfs_iomap.h" +#include "xfs_buf.h" MODULE_AUTHOR("Silicon Graphics, Inc."); MODULE_DESCRIPTION("Additional kdb commands for debugging XFS"); @@ -2350,8 +2351,7 @@ kdbm_bp(int argc, const char **argv) static int kdbm_bpdelay(int argc, const char **argv) { -#ifdef DEBUG - extern struct list_head xfs_buftarg_list; + struct list_head *xfs_buftarg_list = xfs_get_buftarg_list(); struct list_head *curr, *next; xfs_buftarg_t *tp, *n; xfs_buf_t bp; @@ -2372,7 +2372,7 @@ kdbm_bpdelay(int argc, const char **argv } - list_for_each_entry_safe(tp, n, &xfs_buftarg_list, bt_list) { + list_for_each_entry_safe(tp, n, xfs_buftarg_list, bt_list) { list_for_each_safe(curr, next, &tp->bt_delwrite_queue) { addr = (unsigned long)list_entry(curr, xfs_buf_t, b_list); if ((diag = kdb_getarea(bp, addr))) @@ -2388,9 +2388,6 @@ kdbm_bpdelay(int argc, const char **argv } } } -#else - kdb_printf("bt_delwrite_queue inaccessible (non-debug)\n"); -#endif return 0; } From owner-xfs@oss.sgi.com Tue Apr 3 05:01:49 2007 Received: with ECARTIS (v1.0.0; list xfs); Tue, 03 Apr 2007 05:01:56 -0700 (PDT) X-Spam-oss-Status: No, score=3.5 required=5.0 tests=BAYES_99 autolearn=no version=3.2.0-pre1-r499012 Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l33C1lfB004228 for ; Tue, 3 Apr 2007 05:01:48 -0700 Received: by an-out-0708.google.com with SMTP id c5so1529305anc for ; Tue, 03 Apr 2007 05:01:47 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=txHZ6dlfmDTAXzC1JJfxIC4zr23HysF+vBV63kPFJQojyrwYBxC+zarceB7LmuBqtg+GU2SQX9tFi5Miwz6XGDX79fdFvkAzmfEmatq5Ti3j0iqD9L3BDryfSKlBN2uqPNXX6oq/B6rX0trdWSr3ZltvhGKbXgbqyy2rdj5FEPg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=kzkQhrK0HCVnZ5TB83FgwDJEDJKg4Znm2oneaxe2FYT0dFJreyO4NtExZAfNS9pfZYvukd4EUpRjlmra1+0X7VaWc4xuFKNhB16tPPw3gefMiRbv7k7Rksrt3A70pAPDnCkyPdoHXcDXBe+Rls0h+3ic02qiNEV+oj2afLR68+I= Received: by 10.100.91.6 with SMTP id o6mr4212834anb.1175598219092; Tue, 03 Apr 2007 04:03:39 -0700 (PDT) Received: by 10.100.138.16 with HTTP; Tue, 3 Apr 2007 04:03:39 -0700 (PDT) Message-ID: <12fac1030704030403t3ffc3599w5a0191476eb8b865@mail.gmail.com> Date: Tue, 3 Apr 2007 13:03:39 +0200 From: Sencer To: xfs@oss.sgi.com Subject: md/dm devices, barrier support, commodity hardware and data integrity MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-archive-position: 11024 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: alisencer@gmail.com Precedence: bulk X-list: xfs Hello, After reading up on XFS, there are a couple of issues that still seem kind of cloudy to me. I am merely a user of filesystems, so forgive me if some issues seem obvious. If you could confirm/clarify/answer the following issues, it would be very helpful to me. Situation 1) We are currently using XFS on a commodity x86 server with SATA drives (with NCQ) on Debian Etch (Kernel: 2.6.18-3-k7). We are also using Software-Raid1 (mdadm). All partitions except /boot are XFS. If I understand the FAQ and recent ml-discussions right, then 1a) without software-raid, we would enjoy write barrier support, however given that we are using md-devices this is not the case (kern.log confirms this by explicitly stating barrier support is disabled for mdX ...). Did this (barrier support with XFS on md) change in later kernels or is it likely to change in the near (or far) future? (I think I read mentions of md, and some kind of barrier-awareness on the ml, but didn't quite understand what effectively follows from it from a users POV). 1b) Given the current circumstances above, we should disable write cache as suggested in the faq (there are actually UPS's but they've failed before) to reduce the possibility of loosing data. Correct? We did need to do some hard-resets, and had power failure, though as of yet we never had problems with lost data on any xfs partitions, and I'd like to make sure it stays that way. 1c) We have backup strategies in place, so I can live with having a few partly damaged files and restoring them from backups. However I am not sure how we would make sure that we can find out about all such damaged files or if any such files exist ( referring to http://oss.sgi.com/projects/xfs/faq.html#nulls ). Are there tools for finding potential candidates for corruption? I am assuming that there would be a way to find out which files were the most recently touched with the help of the journal. Or do I just use shell-magic and find files by mtime and check if there are Nulls at the end of those file modified within the last minute or two before the crash? Situation 2) I hear many people saying that using XFS on machines that have no UPS (as in Notebooks [battery removed], Desktops etc.) is something that is not recommended. But after reading up on the issues, the recommendation should really go for every FS that only does meta-data journaling, as alluded to in the FAQ. 2a) And with the recent changes (barrier support and sync on truncated+modified+closed files) I assume there is really no reason to choose another meta-data journaling FS over XFS for such machines in terms of likelihood of damaged files after hard-resets and power failures - would you agree? 2b) When dm-crypt+luks is being used, there is no barrier support available (for XFS) even if the underlying hdd supports it, correct? Should this be expected to change, or is it more likely to stay that way? (due to limited dev. resources and priorities? or due to principal issues with it?) Thanks in advance Sencer From owner-xfs@oss.sgi.com Tue Apr 3 12:40:59 2007 Received: with ECARTIS (v1.0.0; list xfs); Tue, 03 Apr 2007 12:41:00 -0700 (PDT) X-Spam-oss-Status: No, score=0.7 required=5.0 tests=AWL,BAYES_60, J_CHICKENPOX_45 autolearn=no version=3.2.0-pre1-r499012 Received: from barcelona.int.jammed.com (jammed.com [216.99.218.161]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l33JevfB027379 for ; Tue, 3 Apr 2007 12:40:58 -0700 Received: from barcelona.int.jammed.com (barcelona.int.jammed.com [172.16.64.15]) by barcelona.int.jammed.com (Postfix) with ESMTP id D5AA2BD6E for ; Tue, 3 Apr 2007 12:11:46 -0700 (PDT) Date: Tue, 3 Apr 2007 12:11:46 -0700 (PDT) From: "James W. Abendschan" X-X-Sender: jwa@barcelona.int.jammed.com To: xfs@oss.sgi.com Subject: xfs_repair segfault Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-archive-position: 11025 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jwa@jammed.com Precedence: bulk X-list: xfs Hi there -- I have a 6.9TB XFS volume that is acting up after a power failure (I understand XFS + no UPS + PC hardware == badness. Not my decision.) The machine is a dual proc x86 (intel xeon 5130) w/ 8GB RAM running a custom 2.6.18 kernel on top of Ubuntu 6.06. Since xfs_check can't repair volumes of this size without scads of memory, I've been using xfs_repair to correct power-related problems before. Unfortunately, for some reason xfs_repair is segfaulting: # ulimit -c unlimited # xfs_repair -v /dev/md1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... zero_log: head block 8 tail block 8 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - agno = 27 - agno = 28 - agno = 29 - agno = 30 - agno = 31 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - clear lost+found (if it exists) ... - clearing existing "lost+found" inode Segmentation fault (core dumped) gdb doesn't show anything useful (I don't know how to interpret the I/O error) : # gdb /sbin/xfs_repair core GNU gdb 6.4-debian Copyright 2005 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i486-linux-gnu"...(no debugging symbols found) Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1". (no debugging symbols found) Core was generated by `xfs_repair -v /dev/md1'. Program terminated with signal 11, Segmentation fault. warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libuuid.so.1 Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found) Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x08052f42 in ?? () (gdb) bt #0 0x08052f42 in ?? () #1 0x000088e9 in ?? () #2 0x00000800 in ?? () #3 0x00000080 in ?? () #4 0x00000000 in ?? () What's the next step? Thanks, James From owner-xfs@oss.sgi.com Tue Apr 3 17:42:40 2007 Received: with ECARTIS (v1.0.0; list xfs); Tue, 03 Apr 2007 17:42:42 -0700 (PDT) X-Spam-oss-Status: No, score=0.3 required=5.0 tests=AWL,BAYES_50, J_CHICKENPOX_45 autolearn=no version=3.2.0-pre1-r499012 Received: from larry.melbourne.sgi.com (larry.melbourne.sgi.com [134.14.52.130]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l340gbfB023972 for ; Tue, 3 Apr 2007 17:42:39 -0700 Received: from pcbnaujok (pc-bnaujok.melbourne.sgi.com [134.14.55.58]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id KAA01820; Wed, 4 Apr 2007 10:42:30 +1000 Message-Id: <200704040042.KAA01820@larry.melbourne.sgi.com> From: "Barry Naujok" To: "'James W. Abendschan'" , Subject: RE: xfs_repair segfault Date: Wed, 4 Apr 2007 10:45:47 +1000 MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 In-Reply-To: Thread-Index: Acd2KC2mjxULtKq3T3C2GrfXBg0FrAAKdumQ X-archive-position: 11026 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: bnaujok@melbourne.sgi.com Precedence: bulk X-list: xfs Hi James, Would it be possible for you apply the patch I posted to xfs@oss in Feb http://oss.sgi.com/archives/xfs/2007-02/msg00072.html to the latest xfsprogs source, make and install it and run: # xfs_metadump /dev/md1 - | bzip2 > /tmp/bad_xfs.bz2 And make the image available for me to download and analyse? Regards, Barry. > -----Original Message----- > From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] > On Behalf Of James W. Abendschan > Sent: Wednesday, 4 April 2007 5:12 AM > To: xfs@oss.sgi.com > Subject: xfs_repair segfault > > Hi there -- I have a 6.9TB XFS volume that is acting up > after a power failure (I understand XFS + no UPS + PC > hardware == badness. Not my decision.) > > The machine is a dual proc x86 (intel xeon 5130) w/ 8GB RAM > running a custom 2.6.18 kernel on top of Ubuntu 6.06. > > Since xfs_check can't repair volumes of this size without > scads of memory, I've been using xfs_repair to correct > power-related problems before. > > Unfortunately, for some reason xfs_repair is segfaulting: > > # ulimit -c unlimited > # xfs_repair -v /dev/md1 > Phase 1 - find and verify superblock... > Phase 2 - using internal log > - zero log... > zero_log: head block 8 tail block 8 > - scan filesystem freespace and inode maps... > - found root inode chunk > Phase 3 - for each AG... > - scan and clear agi unlinked lists... > - process known inodes and perform inode discovery... > - agno = 0 > - agno = 1 > - agno = 2 > - agno = 3 > - agno = 4 > - agno = 5 > - agno = 6 > - agno = 7 > - agno = 8 > - agno = 9 > - agno = 10 > - agno = 11 > - agno = 12 > - agno = 13 > - agno = 14 > - agno = 15 > - agno = 16 > - agno = 17 > - agno = 18 > - agno = 19 > - agno = 20 > - agno = 21 > - agno = 22 > - agno = 23 > - agno = 24 > - agno = 25 > - agno = 26 > - agno = 27 > - agno = 28 > - agno = 29 > - agno = 30 > - agno = 31 > - process newly discovered inodes... > Phase 4 - check for duplicate blocks... > - setting up duplicate extent list... > - clear lost+found (if it exists) ... > - clearing existing "lost+found" inode > Segmentation fault (core dumped) > > > gdb doesn't show anything useful (I don't know how to interpret > the I/O error) : > > > # gdb /sbin/xfs_repair core > GNU gdb 6.4-debian > Copyright 2005 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public > License, and you are > welcome to change it and/or distribute copies of it under > certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show > warranty" for details. > This GDB was configured as "i486-linux-gnu"...(no debugging > symbols found) > Using host libthread_db library > "/lib/tls/i686/cmov/libthread_db.so.1". > > (no debugging symbols found) > Core was generated by `xfs_repair -v /dev/md1'. > Program terminated with signal 11, Segmentation fault. > > warning: Can't read pathname for load map: Input/output error. > Reading symbols from /lib/libuuid.so.1...(no debugging > symbols found)...done. > Loaded symbols for /lib/libuuid.so.1 > Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no > debugging symbols found) > Loaded symbols for /lib/tls/i686/cmov/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...(no debugging > symbols found)...done. > Loaded symbols for /lib/ld-linux.so.2 > > #0 0x08052f42 in ?? () > (gdb) bt > #0 0x08052f42 in ?? () > #1 0x000088e9 in ?? () > #2 0x00000800 in ?? () > #3 0x00000080 in ?? () > #4 0x00000000 in ?? () > > > What's the next step? > > Thanks, > James > > > From owner-xfs@oss.sgi.com Wed Apr 4 06:24:22 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:24:27 -0700 (PDT) X-Spam-oss-Status: No, score=0.8 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DOKfB022642 for ; Wed, 4 Apr 2007 06:24:21 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 373B8BF30 for ; Wed, 4 Apr 2007 15:05:39 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 11168-04 for ; Wed, 4 Apr 2007 15:05:35 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id 3AA7CBF84; Wed, 4 Apr 2007 15:05:35 +0200 (CEST) Date: Wed, 4 Apr 2007 15:05:35 +0200 From: Thomas Kaehn To: xfs@oss.sgi.com Subject: Strange delete performance using XFS Message-ID: <20070404130535.GE18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.5.9i X-archive-position: 11033 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 2042 Lines: 62 Hi, I've got a strange problem on one machine using XFS. Deleting large directories (containing about 100000 files, 20k each) using "rm -rf" lasts nearly as long as creating the the files using a bash loop. The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel. CPU: Dual Xeon(TM) CPU 3.20GHz RAM: 4 GB RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP (Firmware Version = FE9X 3.08.00.004) The XFS was first created using default options and later on with "-d su=64k,sw=2 -l su=64k" which improved overall performance but not delete performance. Has anyone realized similar effects? On a different server (Dell 6850) the directory can be deleted within seconds. What could be the reason for the huge difference in delete performance? Please see below for "time" output. | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done | | real 6m6.814s | user 0m30.290s | sys 2m42.562s | # time rm -rf y | | real 5m18.034s | user 0m0.036s | sys 0m8.169s In contrast to this the result on the Dell machine looks more reasonable: | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done | | real 9m26.658s | user 0m24.134s | sys 3m3.623s | # time rm -rf x | | real 0m10.254s | user 0m0.124s | sys 0m10.105s Ciao, Thomas PS: Using JFS and ext3 it is also possible to delete the above directory in a couple of seconds. Only XFS seems problematic in this regard on this system. -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Wed Apr 4 06:29:49 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:29:55 -0700 (PDT) X-Spam-oss-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DTlfB024034 for ; Wed, 4 Apr 2007 06:29:49 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id B8834F063B21; Wed, 4 Apr 2007 09:29:46 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id B314A7176724; Wed, 4 Apr 2007 09:29:46 -0400 (EDT) Date: Wed, 4 Apr 2007 09:29:46 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404130535.GE18320@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-2087494152-1175693386=:7309" X-archive-position: 11034 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2784 Lines: 86 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-2087494152-1175693386=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi, > > I've got a strange problem on one machine using XFS. Deleting large > directories (containing about 100000 files, 20k each) using "rm -rf" > lasts nearly as long as creating the the files using a bash loop. > > The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel. > CPU: Dual Xeon(TM) CPU 3.20GHz > RAM: 4 GB > RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP > (Firmware Version =3D FE9X 3.08.00.004) > > The XFS was first created using default options and later on with > "-d su=3D64k,sw=3D2 -l su=3D64k" which improved overall performance > but not delete performance. > > Has anyone realized similar effects? On a different server (Dell 6850) > the directory can be deleted within seconds. What could be the reason > for the huge difference in delete performance? > > Please see below for "time" output. > > | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k co= unt=3D20 >/dev/null 2>&1; done > | > | real 6m6.814s > | user 0m30.290s > | sys 2m42.562s > | # time rm -rf y > | > | real 5m18.034s > | user 0m0.036s > | sys 0m8.169s > > In contrast to this the result on the Dell machine looks more > reasonable: > > | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k co= unt=3D20 >/dev/null 2>&1; done > | > | real 9m26.658s > | user 0m24.134s > | sys 3m3.623s > | # time rm -rf x > | > | real 0m10.254s > | user 0m0.124s > | sys 0m10.105s > > Ciao, > Thomas > > PS: Using JFS and ext3 it is also possible to delete the above directory > in a couple of seconds. Only XFS seems problematic in this regard on > this system. > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > > Deletes on XFS is one area that is a little slower than other filesystems.= =20 You can increase the log size during the creation of the filesystem and=20 also increase logbufs to 8 and that might help.= ---1463747160-2087494152-1175693386=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 06:47:31 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:47:38 -0700 (PDT) X-Spam-oss-Status: No, score=0.6 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DlTfB027541 for ; Wed, 4 Apr 2007 06:47:31 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 74F7CC159; Wed, 4 Apr 2007 15:47:29 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 26351-05; Wed, 4 Apr 2007 15:47:24 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id E983AC126; Wed, 4 Apr 2007 15:47:24 +0200 (CEST) Date: Wed, 4 Apr 2007 15:47:24 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070404134724.GF18320@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11035 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 1850 Lines: 49 Hi Justin, On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: > >Please see below for "time" output. > > > >| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 > >>/dev/null 2>&1; done > >| > >| real 6m6.814s > >| user 0m30.290s > >| sys 2m42.562s > >| # time rm -rf y > >| > >| real 5m18.034s > >| user 0m0.036s > >| sys 0m8.169s > Deletes on XFS is one area that is a little slower than other filesystems. > You can increase the log size during the creation of the filesystem and > also increase logbufs to 8 and that might help. Thanks for your suggestions. I also tried to increase the log size and logbufs mount option. This optimizes create and delete times to the above values (with default options both are around 9-10 minutes). The strange thing is that on a similar Dell machines using XFS, too, deletes take only ten seconds which would match user and system time. More than five minutes for deleting 100000 files where ext3 needs 3 seconds on the same machine is actually more than a little bit slower - to my mind there must be something wrong. JFS needs around 18 seconds. However I am not sure if the problem is hardware or software related. I've also tried to use the newest 3ware firmware - but this did not lead to an improvement. Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Wed Apr 4 06:51:14 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:51:20 -0700 (PDT) X-Spam-oss-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DpAfB028519 for ; Wed, 4 Apr 2007 06:51:12 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id 5A323F063B21; Wed, 4 Apr 2007 09:51:10 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 553C57176724; Wed, 4 Apr 2007 09:51:10 -0400 (EDT) Date: Wed, 4 Apr 2007 09:51:10 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404134724.GF18320@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-732358276-1175694670=:7309" X-archive-position: 11036 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2419 Lines: 72 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-732358276-1175694670=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>> Please see below for "time" output. >>> >>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k = count=3D20 >>>> /dev/null 2>&1; done >>> | >>> | real 6m6.814s >>> | user 0m30.290s >>> | sys 2m42.562s >>> | # time rm -rf y >>> | >>> | real 5m18.034s >>> | user 0m0.036s >>> | sys 0m8.169s > >> Deletes on XFS is one area that is a little slower than other filesystem= s. >> You can increase the log size during the creation of the filesystem and >> also increase logbufs to 8 and that might help. > > Thanks for your suggestions. > > I also tried to increase the log size and logbufs mount option. This > optimizes create and delete times to the above values (with default optio= ns > both are around 9-10 minutes). > > The strange thing is that on a similar Dell machines using XFS, too, > deletes take only ten seconds which would match user and system time. > > More than five minutes for deleting 100000 files where ext3 needs > 3 seconds on the same machine is actually more than a little bit slower > - to my mind there must be something wrong. JFS needs around 18 seconds. > > However I am not sure if the problem is hardware or software related. > I've also tried to use the newest 3ware firmware - but this did not lead > to an improvement. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > > I am running some benchmarks with SW raid and will prevent my findings=20 shortly.= ---1463747160-732358276-1175694670=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 06:57:22 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:57:26 -0700 (PDT) X-Spam-oss-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DvKfB030429 for ; Wed, 4 Apr 2007 06:57:21 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id CFFF2F063B21; Wed, 4 Apr 2007 09:57:16 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id CAB0E7176724; Wed, 4 Apr 2007 09:57:16 -0400 (EDT) Date: Wed, 4 Apr 2007 09:57:16 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404134724.GF18320@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-295697546-1175695036=:7309" X-archive-position: 11037 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2757 Lines: 90 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-295697546-1175695036=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>> Please see below for "time" output. >>> >>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k = count=3D20 >>>> /dev/null 2>&1; done >>> | >>> | real 6m6.814s >>> | user 0m30.290s >>> | sys 2m42.562s >>> | # time rm -rf y >>> | >>> | real 5m18.034s >>> | user 0m0.036s >>> | sys 0m8.169s > >> Deletes on XFS is one area that is a little slower than other filesystem= s. >> You can increase the log size during the creation of the filesystem and >> also increase logbufs to 8 and that might help. > > Thanks for your suggestions. > > I also tried to increase the log size and logbufs mount option. This > optimizes create and delete times to the above values (with default optio= ns > both are around 9-10 minutes). > > The strange thing is that on a similar Dell machines using XFS, too, > deletes take only ten seconds which would match user and system time. > > More than five minutes for deleting 100000 files where ext3 needs > 3 seconds on the same machine is actually more than a little bit slower > - to my mind there must be something wrong. JFS needs around 18 seconds. > > However I am not sure if the problem is hardware or software related. > I've also tried to use the newest 3ware firmware - but this did not lead > to an improvement. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > > The benchmark: $ time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k count= =3D20=20 >/dev/null 2>&1; done 1. Six 400GB SATA drives using SW RAID5: real 6m24.411s user 0m43.097s sys 2m17.350s 2. Four Raptor 150 ADFD drives using SW RAID5: real 3m16.962s user 0m42.899s sys 2m15.420s 3. Two Raptor 74GB *GD drives using SW RAID1: real 3m19.241s user 0m41.731s sys 2m15.873s ---1463747160-295697546-1175695036=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 06:57:58 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 06:58:03 -0700 (PDT) X-Spam-oss-Status: No, score=-0.8 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34DvufB030663 for ; Wed, 4 Apr 2007 06:57:57 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id 4A17FF063B21; Wed, 4 Apr 2007 09:57:56 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 269017176724; Wed, 4 Apr 2007 09:57:56 -0400 (EDT) Date: Wed, 4 Apr 2007 09:57:56 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1613578982-1175695076=:7309" X-archive-position: 11038 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2627 Lines: 79 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1613578982-1175695076=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Justin Piszcz wrote: > > > On Wed, 4 Apr 2007, Thomas Kaehn wrote: > >> Hi Justin, >>=20 >> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>>> Please see below for "time" output. >>>>=20 >>>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k= count=3D20 >>>>> /dev/null 2>&1; done >>>> | >>>> | real 6m6.814s >>>> | user 0m30.290s >>>> | sys 2m42.562s >>>> | # time rm -rf y >>>> | >>>> | real 5m18.034s >>>> | user 0m0.036s >>>> | sys 0m8.169s >>=20 >>> Deletes on XFS is one area that is a little slower than other filesyste= ms. >>> You can increase the log size during the creation of the filesystem and >>> also increase logbufs to 8 and that might help. >>=20 >> Thanks for your suggestions. >>=20 >> I also tried to increase the log size and logbufs mount option. This >> optimizes create and delete times to the above values (with default opti= ons >> both are around 9-10 minutes). >>=20 >> The strange thing is that on a similar Dell machines using XFS, too, >> deletes take only ten seconds which would match user and system time. >>=20 >> More than five minutes for deleting 100000 files where ext3 needs >> 3 seconds on the same machine is actually more than a little bit slower >> - to my mind there must be something wrong. JFS needs around 18 seconds. >>=20 >> However I am not sure if the problem is hardware or software related. >> I've also tried to use the newest 3ware firmware - but this did not lead >> to an improvement. >>=20 >> Ciao, >> Thomas >> --=20 >> Thomas K=E4hn WESTEND GmbH | Internet-Business-Provi= der >> Technik CISCO Systems Partner - Authorized Reseller >> Im S=FCsterfeld 6 Tel 0241/701333-= 18 >> tk@westend.com D-52072 Aachen Fax 0241/911879 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 >> Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael= Kolb >>=20 >>=20 > > I am running some benchmarks with SW raid and will prevent my findings=20 > shortly. Removal tests coming shortly, benchmarking is always interesting.= ---1463747160-1613578982-1175695076=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 07:12:51 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 07:12:57 -0700 (PDT) X-Spam-oss-Status: No, score=-0.8 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34ECmfB002491 for ; Wed, 4 Apr 2007 07:12:50 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id 6A9EEF063B21; Wed, 4 Apr 2007 10:12:48 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 655AF7176724; Wed, 4 Apr 2007 10:12:48 -0400 (EDT) Date: Wed, 4 Apr 2007 10:12:48 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1228164866-1175695968=:7309" X-archive-position: 11039 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 3274 Lines: 117 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1228164866-1175695968=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Justin Piszcz wrote: > > > On Wed, 4 Apr 2007, Thomas Kaehn wrote: > >> Hi Justin, >>=20 >> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>>> Please see below for "time" output. >>>>=20 >>>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k= count=3D20 >>>>> /dev/null 2>&1; done >>>> | >>>> | real 6m6.814s >>>> | user 0m30.290s >>>> | sys 2m42.562s >>>> | # time rm -rf y >>>> | >>>> | real 5m18.034s >>>> | user 0m0.036s >>>> | sys 0m8.169s >>=20 >>> Deletes on XFS is one area that is a little slower than other filesyste= ms. >>> You can increase the log size during the creation of the filesystem and >>> also increase logbufs to 8 and that might help. >>=20 >> Thanks for your suggestions. >>=20 >> I also tried to increase the log size and logbufs mount option. This >> optimizes create and delete times to the above values (with default opti= ons >> both are around 9-10 minutes). >>=20 >> The strange thing is that on a similar Dell machines using XFS, too, >> deletes take only ten seconds which would match user and system time. >>=20 >> More than five minutes for deleting 100000 files where ext3 needs >> 3 seconds on the same machine is actually more than a little bit slower >> - to my mind there must be something wrong. JFS needs around 18 seconds. >>=20 >> However I am not sure if the problem is hardware or software related. >> I've also tried to use the newest 3ware firmware - but this did not lead >> to an improvement. >>=20 >> Ciao, >> Thomas >> --=20 >> Thomas K=E4hn WESTEND GmbH | Internet-Business-Provi= der >> Technik CISCO Systems Partner - Authorized Reseller >> Im S=FCsterfeld 6 Tel 0241/701333-= 18 >> tk@westend.com D-52072 Aachen Fax 0241/911879 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 >> Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael= Kolb >>=20 >>=20 > > The benchmark: > $ time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k coun= t=3D20=20 >> /dev/null 2>&1; done > > 1. Six 400GB SATA drives using SW RAID5: > real 6m24.411s > user 0m43.097s > sys 2m17.350s > > 2. Four Raptor 150 ADFD drives using SW RAID5: > real 3m16.962s > user 0m42.899s > sys 2m15.420s > > 3. Two Raptor 74GB *GD drives using SW RAID1: > real 3m19.241s > user 0m41.731s > sys 2m15.873s > > The removals: The benchmark: $ time rm -rf test 1. Six 400GB SATA drives using SW RAID5: real 0m33.996s user 0m0.057s sys 0m8.101s 2. Four Raptor 150 ADFD drives using SW RAID5: real 0m43.967s user 0m0.071s sys 0m8.340s 3. Two Raptor 74GB *GD drives using SW RAID1: real 0m32.965s user 0m0.049s sys 0m6.307s ---1463747160-1228164866-1175695968=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 07:13:39 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 07:13:43 -0700 (PDT) X-Spam-oss-Status: No, score=-0.8 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34EDafB002895 for ; Wed, 4 Apr 2007 07:13:39 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id 55C01F080A24; Wed, 4 Apr 2007 10:13:36 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 50D137176724; Wed, 4 Apr 2007 10:13:36 -0400 (EDT) Date: Wed, 4 Apr 2007 10:13:36 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-2094664383-1175696016=:7309" X-archive-position: 11040 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 3303 Lines: 109 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-2094664383-1175696016=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Justin Piszcz wrote: > > > On Wed, 4 Apr 2007, Thomas Kaehn wrote: > >> Hi Justin, >>=20 >> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>>> Please see below for "time" output. >>>>=20 >>>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k= count=3D20 >>>>> /dev/null 2>&1; done >>>> | >>>> | real 6m6.814s >>>> | user 0m30.290s >>>> | sys 2m42.562s >>>> | # time rm -rf y >>>> | >>>> | real 5m18.034s >>>> | user 0m0.036s >>>> | sys 0m8.169s >>=20 >>> Deletes on XFS is one area that is a little slower than other filesyste= ms. >>> You can increase the log size during the creation of the filesystem and >>> also increase logbufs to 8 and that might help. >>=20 >> Thanks for your suggestions. >>=20 >> I also tried to increase the log size and logbufs mount option. This >> optimizes create and delete times to the above values (with default opti= ons >> both are around 9-10 minutes). >>=20 >> The strange thing is that on a similar Dell machines using XFS, too, >> deletes take only ten seconds which would match user and system time. >>=20 >> More than five minutes for deleting 100000 files where ext3 needs >> 3 seconds on the same machine is actually more than a little bit slower >> - to my mind there must be something wrong. JFS needs around 18 seconds. >>=20 >> However I am not sure if the problem is hardware or software related. >> I've also tried to use the newest 3ware firmware - but this did not lead >> to an improvement. >>=20 >> Ciao, >> Thomas >> --=20 >> Thomas K=E4hn WESTEND GmbH | Internet-Business-Provi= der >> Technik CISCO Systems Partner - Authorized Reseller >> Im S=FCsterfeld 6 Tel 0241/701333-= 18 >> tk@westend.com D-52072 Aachen Fax 0241/911879 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 >> Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael= Kolb >>=20 >>=20 > > The benchmark: > $ time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k coun= t=3D20=20 >> /dev/null 2>&1; done > > 1. Six 400GB SATA drives using SW RAID5: > real 6m24.411s > user 0m43.097s > sys 2m17.350s > > 2. Four Raptor 150 ADFD drives using SW RAID5: > real 3m16.962s > user 0m42.899s > sys 2m15.420s > > 3. Two Raptor 74GB *GD drives using SW RAID1: > real 3m19.241s > user 0m41.731s > sys 2m15.873s > > I used the DEFAULT create options for XFS as I find it highly optimizes=20 itself (at least with SW raid) with the exception of the ROOT FS, I had=20 that optimized awhile ago and I kept it: /dev/md2 / xfs=20 logbufs=3D8,logbsize=3D262144,biosize=3D16,noatime,nodiratime,nobarrier 0 = 1 For my regular RAID5s though I use defaults,noatime. Justin.= ---1463747160-2094664383-1175696016=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 07:22:12 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 07:22:18 -0700 (PDT) X-Spam-oss-Status: No, score=0.5 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34EMBfB005858 for ; Wed, 4 Apr 2007 07:22:12 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id E39C7C16B; Wed, 4 Apr 2007 16:22:10 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 05404-04-13; Wed, 4 Apr 2007 16:22:07 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id 6593AC12D; Wed, 4 Apr 2007 16:21:42 +0200 (CEST) Date: Wed, 4 Apr 2007 16:21:42 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070404142142.GG18320@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11041 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 1210 Lines: 37 Hi Justin, On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote: > On Wed, 4 Apr 2007, Justin Piszcz wrote: > >On Wed, 4 Apr 2007, Thomas Kaehn wrote: > >$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 > >>/dev/null 2>&1; done > > > >1. Six 400GB SATA drives using SW RAID5: > >real 6m24.411s > >user 0m43.097s > >sys 2m17.350s > > > > The removals: > The benchmark: > $ time rm -rf test > > 1. Six 400GB SATA drives using SW RAID5: > real 0m33.996s > user 0m0.057s > sys 0m8.101s thanks for your bechmark. To my mind this clearly shows that my setup is wrong at some point. I'll try again with your mount options. Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Wed Apr 4 07:24:46 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 07:24:50 -0700 (PDT) X-Spam-oss-Status: No, score=-0.7 required=5.0 tests=AWL,BAYES_50, J_CHICKENPOX_43 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34EOhfB006635 for ; Wed, 4 Apr 2007 07:24:45 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id A39DAF063B21; Wed, 4 Apr 2007 10:24:42 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 9FC3C717672A; Wed, 4 Apr 2007 10:24:42 -0400 (EDT) Date: Wed, 4 Apr 2007 10:24:42 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404142142.GG18320@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> <20070404142142.GG18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-702127020-1175696682=:7309" X-archive-position: 11042 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 1917 Lines: 63 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-702127020-1175696682=:7309 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote: >> On Wed, 4 Apr 2007, Justin Piszcz wrote: >>> On Wed, 4 Apr 2007, Thomas Kaehn wrote: >>> $ time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k co= unt=3D20 >>>> /dev/null 2>&1; done >>> >>> 1. Six 400GB SATA drives using SW RAID5: >>> real 6m24.411s >>> user 0m43.097s >>> sys 2m17.350s >>> >> >> The removals: >> The benchmark: >> $ time rm -rf test >> >> 1. Six 400GB SATA drives using SW RAID5: >> real 0m33.996s >> user 0m0.057s >> sys 0m8.101s > > thanks for your bechmark. To my mind this clearly shows that my > setup is wrong at some point. I'll try again with your mount options. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > > My guess is mkfs.xfs cannot optimzie for your array like it can with a SW= =20 RAID device because it cannot see what is undereath it. Have you tried=20 making a SW RAID? I also use optimized parameters for my SW RAID1/5 as=20 well FYI. Justin.= ---1463747160-702127020-1175696682=:7309-- From owner-xfs@oss.sgi.com Wed Apr 4 07:35:47 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 07:35:52 -0700 (PDT) X-Spam-oss-Status: No, score=0.5 required=5.0 tests=AWL,BAYES_50, J_CHICKENPOX_43 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34EZkfB008940 for ; Wed, 4 Apr 2007 07:35:47 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 387EDC177; Wed, 4 Apr 2007 16:35:45 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 10353-03-2; Wed, 4 Apr 2007 16:35:37 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id B287BC117; Wed, 4 Apr 2007 16:35:33 +0200 (CEST) Date: Wed, 4 Apr 2007 16:35:33 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070404143533.GF12481@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> <20070404142142.GG18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11043 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 1933 Lines: 53 Hi Justin, On Wed, Apr 04, 2007 at 10:24:42AM -0400, Justin Piszcz wrote: > >On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote: > >>On Wed, 4 Apr 2007, Justin Piszcz wrote: > >>>On Wed, 4 Apr 2007, Thomas Kaehn wrote: > >>>$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 > >>>>/dev/null 2>&1; done > >>> > My guess is mkfs.xfs cannot optimzie for your array like it can with a SW > RAID device because it cannot see what is undereath it. Have you tried > making a SW RAID? I also use optimized parameters for my SW RAID1/5 as > well FYI. I guess this might be the problem. I've already tried to alter the stripe unit to match the RAID stripe size: "-d su=64k,sw=2 -l su=64k". Maybe the 3ware controller can't deal with the kind of read and write patterns needed by XFS. But in this case other people should have realized similar problems. On a different system with a 3ware 9500S-4LP using 4 disks as RAID5 setup I get a better (but not really good) result for delete performance (I've taken only 50000 files in this case as the system's CPU is much slower): | # time for i in `seq 1 50000`; do dd if=/dev/zero of=$i | bs=1k count=20 >/dev/null 2>&1; done | | real 18m21.643s | user 0m55.727s | sys 3m12.140s | backup:/srv/x# cd .. | backup:/srv# rm -rf x | | # time rm -rf x | | real 5m7.845s | user 0m0.160s | sys 0m11.369s Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Wed Apr 4 08:45:29 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 08:45:33 -0700 (PDT) X-Spam-oss-Status: No, score=-0.4 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from smtp108.sbc.mail.mud.yahoo.com (smtp108.sbc.mail.mud.yahoo.com [68.142.198.207]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id l34FjRfB026359 for ; Wed, 4 Apr 2007 08:45:28 -0700 Received: (qmail 95791 invoked from network); 4 Apr 2007 15:45:26 -0000 Received: from unknown (HELO stupidest.org) (cwedgwood@sbcglobal.net@24.5.75.45 with login) by smtp108.sbc.mail.mud.yahoo.com with SMTP; 4 Apr 2007 15:45:25 -0000 X-YMail-OSG: VVG0PLEVM1mC181ic3eeqjwe.pA93jXFNrc6sdqfX4N8N.o06NwfUPyuHOiCpMxcsJnKi4Utiw-- Received: by tuatara.stupidest.org (Postfix, from userid 10000) id 74C8D1826127; Wed, 4 Apr 2007 08:45:23 -0700 (PDT) Date: Wed, 4 Apr 2007 08:45:23 -0700 From: Chris Wedgwood To: Thomas Kaehn Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070404154523.GA20096@tuatara.stupidest.org> References: <20070404130535.GE18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070404130535.GE18320@mail3b.westend.com> X-archive-position: 11044 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: cw@f00f.org Precedence: bulk X-list: xfs Content-Length: 1390 Lines: 46 On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote: > I've got a strange problem on one machine using XFS. Deleting large > directories (containing about 100000 files, 20k each) using "rm -rf" > lasts nearly as long as creating the the files using a bash loop. quite possible > RAM: 4 GB > RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP > (Firmware Version = FE9X 3.08.00.004) > The XFS was first created using default options and later on with > "-d su=64k,sw=2 -l su=64k" which improved overall performance > but not delete performance. have you tried w/o using the hw raid? > Has anyone realized similar effects? On a different server (Dell > 6850) the directory can be deleted within seconds. What could be the > reason for the huge difference in delete performance? a lot of log updates; does the other server have a battery-backed write-cache like many cards to these days? > | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done > | > | real 6m6.814s > | user 0m30.290s > | sys 2m42.562s that's about the same as my quick single-spindle cheap-desktop test here > | # time rm -rf y > | > | real 5m18.034s > | user 0m0.036s > | sys 0m8.169s v2 logs? what logbufs & logbsize is used? testing with my cheap crappy desktop workstation thing with a single disk I get "1m25.004s" for the delete From owner-xfs@oss.sgi.com Wed Apr 4 11:36:41 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 11:36:54 -0700 (PDT) X-Spam-oss-Status: No, score=-0.7 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34IadfB001401 for ; Wed, 4 Apr 2007 11:36:40 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id 8B561F063B21; Wed, 4 Apr 2007 14:36:37 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id 8A4A4717672A; Wed, 4 Apr 2007 14:36:37 -0400 (EDT) Date: Wed, 4 Apr 2007 14:36:37 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404134724.GF18320@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1108556179-1175711797=:16731" X-archive-position: 11045 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2388 Lines: 70 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1108556179-1175711797=:16731 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote: >>> Please see below for "time" output. >>> >>> | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k = count=3D20 >>>> /dev/null 2>&1; done >>> | >>> | real 6m6.814s >>> | user 0m30.290s >>> | sys 2m42.562s >>> | # time rm -rf y >>> | >>> | real 5m18.034s >>> | user 0m0.036s >>> | sys 0m8.169s > >> Deletes on XFS is one area that is a little slower than other filesystem= s. >> You can increase the log size during the creation of the filesystem and >> also increase logbufs to 8 and that might help. > > Thanks for your suggestions. > > I also tried to increase the log size and logbufs mount option. This > optimizes create and delete times to the above values (with default optio= ns > both are around 9-10 minutes). > > The strange thing is that on a similar Dell machines using XFS, too, > deletes take only ten seconds which would match user and system time. > > More than five minutes for deleting 100000 files where ext3 needs > 3 seconds on the same machine is actually more than a little bit slower > - to my mind there must be something wrong. JFS needs around 18 seconds. > > However I am not sure if the problem is hardware or software related. > I've also tried to use the newest 3ware firmware - but this did not lead > to an improvement. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > For the ext3, try time bash -c 'rm -rf test; sync' ---1463747160-1108556179-1175711797=:16731-- From owner-xfs@oss.sgi.com Wed Apr 4 13:45:08 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 13:45:12 -0700 (PDT) X-Spam-oss-Status: No, score=-0.6 required=5.0 tests=AWL,BAYES_50, J_CHICKENPOX_43 autolearn=no version=3.2.0-pre1-r499012 Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34Kj4fB005320 for ; Wed, 4 Apr 2007 13:45:08 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id D56A6F063B21; Wed, 4 Apr 2007 16:45:03 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id D21F9717672A; Wed, 4 Apr 2007 16:45:03 -0400 (EDT) Date: Wed, 4 Apr 2007 16:45:03 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070404143533.GF12481@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> <20070404142142.GG18320@mail3b.westend.com> <20070404143533.GF12481@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1667215911-1175719503=:20373" X-archive-position: 11046 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs Content-Length: 2514 Lines: 73 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1667215911-1175719503=:20373 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 4 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Wed, Apr 04, 2007 at 10:24:42AM -0400, Justin Piszcz wrote: >>> On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote: >>>> On Wed, 4 Apr 2007, Justin Piszcz wrote: >>>>> On Wed, 4 Apr 2007, Thomas Kaehn wrote: >>>>> $ time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k = count=3D20 >>>>>> /dev/null 2>&1; done >>>>> >> My guess is mkfs.xfs cannot optimzie for your array like it can with a SW >> RAID device because it cannot see what is undereath it. Have you tried >> making a SW RAID? I also use optimized parameters for my SW RAID1/5 as >> well FYI. > > I guess this might be the problem. I've already tried to alter > the stripe unit to match the RAID stripe size: "-d su=3D64k,sw=3D2 -l su= =3D64k". > > Maybe the 3ware controller can't deal with the kind of read and write > patterns needed by XFS. But in this case other people should have > realized similar problems. > > On a different system with a 3ware 9500S-4LP using 4 disks as RAID5 > setup I get a better (but not really good) result for delete > performance (I've taken only 50000 files in this case as the system's > CPU is much slower): > > | # time for i in `seq 1 50000`; do dd if=3D/dev/zero of=3D$i > | bs=3D1k count=3D20 >/dev/null 2>&1; done > | > | real 18m21.643s > | user 0m55.727s > | sys 3m12.140s > | backup:/srv/x# cd .. > | backup:/srv# rm -rf x > | > | # time rm -rf x > | > | real 5m7.845s > | user 0m0.160s > | sys 0m11.369s > > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > What do you get with ext3 when using time bash -c 'rm -f file; sync'= ---1463747160-1667215911-1175719503=:20373-- From owner-xfs@oss.sgi.com Wed Apr 4 13:58:27 2007 Received: with ECARTIS (v1.0.0; list xfs); Wed, 04 Apr 2007 13:58:30 -0700 (PDT) X-Spam-oss-Status: No, score=0.6 required=5.0 tests=BAYES_50,J_CHICKENPOX_43, J_CHICKENPOX_44,J_CHICKENPOX_45,J_CHICKENPOX_46,J_CHICKENPOX_47, J_CHICKENPOX_48 autolearn=no version=3.2.0-pre1-r499012 Received: from mail.linbit.com (nudl.linbit.com [212.69.162.21]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l34KwPfB008103 for ; Wed, 4 Apr 2007 13:58:27 -0700 Received: by mail.linbit.com (LINBIT Mail Daemon, from userid 1030) id 7AE342E10D59; Wed, 4 Apr 2007 22:36:06 +0200 (CEST) Date: Wed, 4 Apr 2007 22:36:01 +0200 X-OfflineIMAP-1301118847-6c61727340696d61702e6c696e626974-494e424f582e4f7574626f78: 1175718968-0570832815641-v4.0.11 From: Lars Ellenberg To: xfs@oss.sgi.com Subject: xfs_repair leaves empty but undeletable dirs in lost+found Message-ID: <20070404203601.GA11771@barkeeper1.linbit> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.11 X-archive-position: 11047 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: lars.ellenberg@linbit.com Precedence: bulk X-list: xfs Content-Length: 9501 Lines: 343 this is a plain debian sarge system, i386 (actually k7 athlon), nothing fancy. kernel is a self-build kernel.org 2.6.16 kernel, repackaged for sarge, so it is no longer obvious which exact sublevel. I can dig that up however, if it seems relevant. this is a backup volume with typically many (40 to 70, maybe?) hardlinks. Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg00-backup 1.2G 19M 1.2G 2% /mnt/backup Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-backup 1.2T 809G 365G 69% /mnt/backup somehow file system corruption crept in. may be related to some power loss, may be related to some strange 3w-xxxx: scsi0: AEN: WARNING: ATA UDMA upgrade: Port #3 messages (also for port 5), which are actually DOWNgrades... (unfortunately 2.6.16 has that double definition bug still in some header file, I'll submit the oneline patch to Adrian together whith some other stuff I have pending). maybe cosmic rays. memtest86+ is fine, btw. anyhow. after several runs of xfs_repair version 2.6.20 (and then cleaning up lost+found, where possible), some "empty" but not so empty directories stay behind. [root:/mnt/backup/lost+found]# rmdir 4059295137 rmdir: `4059295137': Directory not empty [root:/mnt/backup/lost+found]# find 4059295137 -ls 4059295137 8 drwxrwxr-x 2 1049 1049 4096 Apr 3 12:39 4059295137 that is it! [right now I have still 118 of these] additional runs of xfs_repair don't change the situation. on run takes about 8+ hours, though. NOTE that I used the default sarge xfsprogs version 2.6.20, not the upstream 2.8.20. yet. I'll start an xfs_repair run with 2.8.20 right after this post, though... from a post on xfs mailing list Message-Id: <200608150145.LAA07105@larry.melbourne.sgi.com> From: Barry Naujok To: 'Paul Slootman' , xfs@oss.sgi.com Subject: RE: cache_purge: shake on cache 0x5880a0 left 8 nodes!? Date: Tue, 15 Aug 2006 11:49:13 +1000 I found a suggestion to investigate like this: xfs_db version 2.6.20 xfs_db /dev/vg00/backup [sorry, I skipped the blockget -n and therefor the ncheck, if absolutely necessary, I can do that still, but I suspect it will take hours, too? I tried it, but after a few minutes with 95% CPU and about 2GB RAM used, I killed it...] xfs_db> inode 4059295137 xfs_db> p core.magic = 0x494e core.mode = 040775 core.version = 1 core.format = 2 (extents) core.nlinkv1 = 2 core.uid = 1049 core.gid = 1049 core.flushiter = 24 core.atime.sec = Thu Feb 1 20:30:54 2007 core.atime.nsec = 533550800 core.mtime.sec = Tue Apr 3 12:39:00 2007 core.mtime.nsec = 472123752 core.ctime.sec = Tue Apr 3 12:39:00 2007 core.ctime.nsec = 472123752 core.size = 4096 core.nblocks = 2 core.extsize = 0 core.nextents = 2 core.naextents = 0 core.forkoff = 0 core.aformat = 2 (extents) core.dmevmask = 0 core.dmstate = 0 core.newrtbm = 0 core.prealloc = 0 core.realtime = 0 core.immutable = 0 core.append = 0 core.sync = 0 core.noatime = 0 core.nodump = 0 core.gen = 11 next_unlinked = null u.bmx[0-1] = [startoff,startblock,blockcount,extentflag] 0:[0,253808988,1,0] 1:[8388608,253808989,1,0] xfs_db> ncheck lost+found must run blockget -n first xfs_db> dblock 0 xfs_db> p dhdr.magic = 0x58443244 dhdr.bestfree[0].offset = 0x20 dhdr.bestfree[0].length = 0xfd0 dhdr.bestfree[1].offset = 0 dhdr.bestfree[1].length = 0 dhdr.bestfree[2].offset = 0 dhdr.bestfree[2].length = 0 du[0].inumber = 4059295137 du[0].namelen = 1 du[0].name = "." du[0].tag = 0x10 du[1].freetag = 0xffff du[1].length = 0xfd0 du[1].tag = 0x20 du[2].inumber = 656 du[2].namelen = 2 du[2].name = ".." du[2].tag = 0xff0 xfs_db> dblock 8388608 xfs_db> p lhdr.info.forw = 0 lhdr.info.back = 0 lhdr.info.magic = 0xd2f1 lhdr.count = 99 lhdr.stale = 97 lbests[0] = 0:0xfd0 lents[0].hashval = 0x2e lents[0].address = 0x2 lents[1].hashval = 0x172e lents[1].address = 0x1fe lents[2].hashval = 0x859d16c lents[2].address = 0 lents[3].hashval = 0xdc6133e lents[3].address = 0 lents[4].hashval = 0xeffc248 lents[4].address = 0 lents[5].hashval = 0xfed728e lents[5].address = 0 lents[6].hashval = 0x124f4f36 lents[6].address = 0 lents[7].hashval = 0x13625491 lents[7].address = 0 lents[8].hashval = 0x1372549d lents[8].address = 0 lents[9].hashval = 0x19ef9ac0 lents[9].address = 0 lents[10].hashval = 0x1b1d6dce lents[10].address = 0 lents[11].hashval = 0x1db2dd93 lents[11].address = 0 lents[12].hashval = 0x262a70f3 lents[12].address = 0 lents[13].hashval = 0x29460811 lents[13].address = 0 lents[14].hashval = 0x2956081d lents[14].address = 0 lents[15].hashval = 0x2d8b48ab lents[15].address = 0 lents[16].hashval = 0x31e3c314 lents[16].address = 0 lents[17].hashval = 0x3669d1ab lents[17].address = 0 lents[18].hashval = 0x3679d1a7 lents[18].address = 0 lents[19].hashval = 0x36b8264b lents[19].address = 0 lents[20].hashval = 0x3996aced lents[20].address = 0 lents[21].hashval = 0x3a35ab00 lents[21].address = 0 lents[22].hashval = 0x3c670bbd lents[22].address = 0 lents[23].hashval = 0x3f150fd1 lents[23].address = 0 lents[24].hashval = 0x4308c7cd lents[24].address = 0 lents[25].hashval = 0x463489b9 lents[25].address = 0 lents[26].hashval = 0x475261d5 lents[26].address = 0 lents[27].hashval = 0x53469be7 lents[27].address = 0 lents[28].hashval = 0x53569beb lents[28].address = 0 lents[29].hashval = 0x589604ba lents[29].address = 0 lents[30].hashval = 0x594bf53d lents[30].address = 0 lents[31].hashval = 0x5981bb01 lents[31].address = 0 lents[32].hashval = 0x5a46bb9c lents[32].address = 0 lents[33].hashval = 0x5c0b668e lents[33].address = 0 lents[34].hashval = 0x5deca0a1 lents[34].address = 0 lents[35].hashval = 0x5f872eba lents[35].address = 0 lents[36].hashval = 0x5f8a3faa lents[36].address = 0 lents[37].hashval = 0x5f9a3fa6 lents[37].address = 0 lents[38].hashval = 0x691b5828 lents[38].address = 0 lents[39].hashval = 0x69d27a70 lents[39].address = 0 lents[40].hashval = 0x73c859c2 lents[40].address = 0 lents[41].hashval = 0x73d859ce lents[41].address = 0 lents[42].hashval = 0x75eb5177 lents[42].address = 0 lents[43].hashval = 0x75fb517b lents[43].address = 0 lents[44].hashval = 0x891e2e5b lents[44].address = 0 lents[45].hashval = 0x8965893f lents[45].address = 0 lents[46].hashval = 0x8cb8b142 lents[46].address = 0 lents[47].hashval = 0x8cce44a0 lents[47].address = 0 lents[48].hashval = 0x8de83f12 lents[48].address = 0 lents[49].hashval = 0x8f22b70e lents[49].address = 0 lents[50].hashval = 0x8f4ab64c lents[50].address = 0 lents[51].hashval = 0x90acb7a0 lents[51].address = 0 lents[52].hashval = 0x9302cac0 lents[52].address = 0 lents[53].hashval = 0x9312cacc lents[53].address = 0 lents[54].hashval = 0x950c4daa lents[54].address = 0 lents[55].hashval = 0x960164df lents[55].address = 0 lents[56].hashval = 0x97df8a76 lents[56].address = 0 lents[57].hashval = 0x9b1ca984 lents[57].address = 0 lents[58].hashval = 0x9be7549e lents[58].address = 0 lents[59].hashval = 0x9bf75492 lents[59].address = 0 lents[60].hashval = 0x9db319d9 lents[60].address = 0 lents[61].hashval = 0xa4926d71 lents[61].address = 0 lents[62].hashval = 0xa5677d7e lents[62].address = 0 lents[63].hashval = 0xa5777d72 lents[63].address = 0 lents[64].hashval = 0xaa01d033 lents[64].address = 0 lents[65].hashval = 0xaa11d03f lents[65].address = 0 lents[66].hashval = 0xaa8a84f0 lents[66].address = 0 lents[67].hashval = 0xaa9a84fc lents[67].address = 0 lents[68].hashval = 0xae810ff0 lents[68].address = 0 lents[69].hashval = 0xaedc9085 lents[69].address = 0 lents[70].hashval = 0xb55cb496 lents[70].address = 0 lents[71].hashval = 0xb8866b93 lents[71].address = 0 lents[72].hashval = 0xb98b5af9 lents[72].address = 0 lents[73].hashval = 0xb9d6e5c3 lents[73].address = 0 lents[74].hashval = 0xbb1c6ddf lents[74].address = 0 lents[75].hashval = 0xbc34bda9 lents[75].address = 0 lents[76].hashval = 0xbd20f48a lents[76].address = 0 lents[77].hashval = 0xbe8bae2a lents[77].address = 0 lents[78].hashval = 0xc86644bc lents[78].address = 0 lents[79].hashval = 0xcdb62e47 lents[79].address = 0 lents[80].hashval = 0xcdcd8923 lents[80].address = 0 lents[81].hashval = 0xd43269bc lents[81].address = 0 lents[82].hashval = 0xde074636 lents[82].address = 0 lents[83].hashval = 0xde17463a lents[83].address = 0 lents[84].hashval = 0xe1321219 lents[84].address = 0 lents[85].hashval = 0xe1465aa0 lents[85].address = 0 lents[86].hashval = 0xe65770a0 lents[86].address = 0 lents[87].hashval = 0xe7e6d19e lents[87].address = 0 lents[88].hashval = 0xeb8d1d7a lents[88].address = 0 lents[89].hashval = 0xf02bb092 lents[89].address = 0 lents[90].hashval = 0xf03bb09e lents[90].address = 0 lents[91].hashval = 0xf494b02c lents[91].address = 0 lents[92].hashval = 0xf58811b2 lents[92].address = 0 lents[93].hashval = 0xf59811be lents[93].address = 0 lents[94].hashval = 0xf988f496 lents[94].address = 0 lents[95].hashval = 0xfb4d59cd lents[95].address = 0 lents[96].hashval = 0xfb5d59c1 lents[96].address = 0 lents[97].hashval = 0xfca78996 lents[97].address = 0 lents[98].hashval = 0xfe80e968 lents[98].address = 0 ltail.bestcount = 1 xfs_db> hope that helps someone figure out what is wrong. if I can provide further info, anything, just tell me. cheers, -- : Lars Ellenberg Tel +43-1-8178292-0 : : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : : Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com : __ please use the "List-Reply" function of your email client. From owner-xfs@oss.sgi.com Thu Apr 5 00:28:08 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 00:28:11 -0700 (PDT) X-Spam-oss-Status: No, score=0.4 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l357S6fB028589 for ; Thu, 5 Apr 2007 00:28:07 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 994E3C308; Thu, 5 Apr 2007 09:28:05 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 25356-05; Thu, 5 Apr 2007 09:28:03 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id 93425C2BC; Thu, 5 Apr 2007 09:28:03 +0200 (CEST) Date: Thu, 5 Apr 2007 09:28:03 +0200 From: Thomas Kaehn To: Chris Wedgwood Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070405072803.GB2759@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404154523.GA20096@tuatara.stupidest.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20070404154523.GA20096@tuatara.stupidest.org> User-Agent: Mutt/1.5.9i X-archive-position: 11049 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 2790 Lines: 71 Hi Chris, On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote: > On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote: > > The XFS was first created using default options and later on with > > "-d su=64k,sw=2 -l su=64k" which improved overall performance > > but not delete performance. > > have you tried w/o using the hw raid? I'am going to test with a single disk in the same machine. > > Has anyone realized similar effects? On a different server (Dell > > 6850) the directory can be deleted within seconds. What could be the > > reason for the huge difference in delete performance? > > a lot of log updates; does the other server have a battery-backed > write-cache like many cards to these days? The Dell system has got a battery-backed write-cache. The 3ware system has no battery unit. However it's supposed to provide write cache, too. At least I've enabled it in the RAID's configuration. The controller has got more than 100MB memory. > > | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done > > | > > | real 6m6.814s > > | user 0m30.290s > > | sys 2m42.562s > > that's about the same as my quick single-spindle cheap-desktop test > here > > > | # time rm -rf y > > | > > | real 5m18.034s > > | user 0m0.036s > > | sys 0m8.169s > > v2 logs? what logbufs & logbsize is used? > > testing with my cheap crappy desktop workstation thing with a > single disk I get "1m25.004s" for the delete Your delete time sounds more sensible to me. The file system was created first with default options - later on I tried to match the RAID's stripe size, increased the log size and mounted with logbufs=8: log stripe unit specified, using v2 logs meta-data=/dev/sda1 isize=256 agcount=8, agsize=125008 blks = sectsz=512 data = bsize=4096 blocks=1000032, imaxpct=25 = sunit=16 swidth=32 blks, unwritten=1 naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=16384, version=2 = sectsz=512 sunit=16 blks realtime =none extsz=65536 blocks=0, rtextents=0 Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Thu Apr 5 00:37:25 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 00:37:27 -0700 (PDT) X-Spam-oss-Status: No, score=0.3 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l357bOfB030945 for ; Thu, 5 Apr 2007 00:37:25 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id A4D40C2EE; Thu, 5 Apr 2007 09:37:23 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 28182-02; Thu, 5 Apr 2007 09:37:21 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id EEA00C2E0; Thu, 5 Apr 2007 09:37:21 +0200 (CEST) Date: Thu, 5 Apr 2007 09:37:21 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070405073721.GC2759@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11050 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 751 Lines: 23 Hi Justin, On Wed, Apr 04, 2007 at 02:36:37PM -0400, Justin Piszcz wrote: > For the ext3, try time bash -c 'rm -rf test; sync' # time bash -c 'rm -rf y; sync' real 0m1.592s user 0m0.032s sys 0m1.408s Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Thu Apr 5 01:17:55 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 01:17:57 -0700 (PDT) X-Spam-oss-Status: No, score=0.3 required=5.0 tests=AWL,BAYES_50 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l358HrfB010719 for ; Thu, 5 Apr 2007 01:17:55 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 66206C17E; Thu, 5 Apr 2007 10:17:53 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 07235-04-3; Thu, 5 Apr 2007 10:17:51 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id 795E4C117; Thu, 5 Apr 2007 10:17:51 +0200 (CEST) Date: Thu, 5 Apr 2007 10:17:51 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070405081751.GD2759@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404134724.GF18320@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11051 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 1654 Lines: 46 Hi Justin, On Wed, Apr 04, 2007 at 10:13:36AM -0400, Justin Piszcz wrote: > I used the DEFAULT create options for XFS as I find it highly optimizes > itself (at least with SW raid) with the exception of the ROOT FS, I had > that optimized awhile ago and I kept it: > > /dev/md2 / xfs > logbufs=8,logbsize=262144,biosize=16,noatime,nodiratime,nobarrier 0 1 > > > For my regular RAID5s though I use defaults,noatime. I've disabled barriers, too, and performance increased dramatically. However I am not aware of the consequences of disabling write barriers. The FAQ generally recommends using write barriers except when having a battery-backed cache (this 3ware has not). | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done | | real 3m52.182s | user 0m30.482s | sys 3m16.152s | | # time \rm -rf y | | real 0m16.327s | user 0m0.052s | sys 0m8.305s So I am unsure if disabling is an option for me. I could imagine that write barriers are not properly supported by 3ware or have to be fine tuned at the kernel or SCSI level. Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Thu Apr 5 01:30:32 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 01:30:35 -0700 (PDT) X-Spam-oss-Status: No, score=0.6 required=5.0 tests=AWL,BAYES_60,HTML_MESSAGE, RDNS_NONE autolearn=no version=3.2.0-pre1-r499012 Received: from ilsmtp.nds.com ([192.118.32.12]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l358UQfD013932 for ; Thu, 5 Apr 2007 01:30:31 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 MIME-Version: 1.0 Subject: XFS Resiliency to the disk errors. Date: Thu, 5 Apr 2007 11:08:07 +0300 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: XFS Resiliency to the disk errors. Thread-Index: Acd3WYs9KYsrse46SLGHxEpphOjGKg== From: "Zak, Semion" To: Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 7bit X-archive-position: 11052 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: SZak@nds.com Precedence: bulk X-list: xfs Content-Length: 481 Lines: 21 Hi, We are studying possibility to use XFS with cheap (not too reliable) discs, so we have some questions: What in XFS is done to survive the disk errors (bad sectors)? I know about superblock duplication in every AG. What else? What is XFS behavior in case of the disk errors (panic/no mount/partial data access)? What could be done to restore? If zero bad sector/dump to other device/format/restore will help? Thanks, Semion. [[HTML alternate version deleted]] From owner-xfs@oss.sgi.com Thu Apr 5 02:03:38 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 02:03:40 -0700 (PDT) X-Spam-oss-Status: No, score=0.4 required=5.0 tests=AWL,BAYES_50, J_CHICKENPOX_12,J_CHICKENPOX_43 autolearn=no version=3.2.0-pre1-r499012 Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l3593afB021908 for ; Thu, 5 Apr 2007 02:03:38 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 0DF9FC14C; Thu, 5 Apr 2007 11:03:36 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 20480-03; Thu, 5 Apr 2007 11:03:33 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id C3122BED3; Thu, 5 Apr 2007 11:03:33 +0200 (CEST) Date: Thu, 5 Apr 2007 11:03:33 +0200 From: Thomas Kaehn To: Chris Wedgwood Cc: xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070405090333.GE2759@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404154523.GA20096@tuatara.stupidest.org> <20070405072803.GB2759@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20070405072803.GB2759@mail3b.westend.com> User-Agent: Mutt/1.5.9i X-archive-position: 11053 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Content-Length: 1421 Lines: 40 Hi Chris, On Thu, Apr 05, 2007 at 09:28:03AM +0200, Thomas Kaehn wrote: > On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote: > > On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote: > > > The XFS was first created using default options and later on with > > > "-d su=64k,sw=2 -l su=64k" which improved overall performance > > > but not delete performance. > > > > have you tried w/o using the hw raid? > > I'am going to test with a single disk in the same machine. this is what I got with a single disk (defaults for mkfs.xfs, logbufs=8 for mount) in the same machine: | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done | | real 11m22.487s | user 0m30.278s | sys 2m33.762s | # time \rm -rf y | | real 8m20.963s | user 0m0.056s | sys 0m7.968s So there is no improvement for a single disk. Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Thu Apr 5 03:21:45 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 03:21:49 -0700 (PDT) Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l35ALifB006633 for ; Thu, 5 Apr 2007 03:21:45 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id BB6C8F063B21; Thu, 5 Apr 2007 06:21:43 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id B3139717672A; Thu, 5 Apr 2007 06:21:43 -0400 (EDT) Date: Thu, 5 Apr 2007 06:21:43 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: Chris Wedgwood , xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070405090333.GE2759@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404154523.GA20096@tuatara.stupidest.org> <20070405072803.GB2759@mail3b.westend.com> <20070405090333.GE2759@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1411685764-1175768503=:17700" X-archive-position: 11054 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1411685764-1175768503=:17700 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Thu, 5 Apr 2007, Thomas Kaehn wrote: > Hi Chris, > > On Thu, Apr 05, 2007 at 09:28:03AM +0200, Thomas Kaehn wrote: >> On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote: >>> On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote: >>>> The XFS was first created using default options and later on with >>>> "-d su=3D64k,sw=3D2 -l su=3D64k" which improved overall performance >>>> but not delete performance. >>> >>> have you tried w/o using the hw raid? >> >> I'am going to test with a single disk in the same machine. > > this is what I got with a single disk (defaults for mkfs.xfs, logbufs=3D8= for > mount) in the same machine: > > | # time for i in `seq 1 100000`; do dd if=3D/dev/zero of=3D$i bs=3D1k co= unt=3D20 >/dev/null 2>&1; done > | > | real 11m22.487s > | user 0m30.278s > | sys 2m33.762s > | # time \rm -rf y > | > | real 8m20.963s > | user 0m0.056s > | sys 0m7.968s > > So there is no improvement for a single disk. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 > Gesch=E4ftsf=FChrer: Thomas Neugebauer, Thomas Heller, Michael = Kolb > > What kind of disks are you using? Maybe just slow disks??= ---1463747160-1411685764-1175768503=:17700-- From owner-xfs@oss.sgi.com Thu Apr 5 03:50:13 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 03:50:17 -0700 (PDT) Received: from mail3b1.westend.com (mail3b.westend.com [212.117.79.78]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l35AoCfB013090 for ; Thu, 5 Apr 2007 03:50:13 -0700 Received: from localhost (localhost [127.0.0.1]) by mail3b1.westend.com (Postfix) with ESMTP id 92453C03B; Thu, 5 Apr 2007 12:50:08 +0200 (CEST) Received: from mail3b1.westend.com ([127.0.0.1]) by localhost (mail3b [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 17632-05; Thu, 5 Apr 2007 12:50:06 +0200 (CEST) Received: by mail3b1.westend.com (Postfix, from userid 1001) id 14D6FBFBF; Thu, 5 Apr 2007 12:50:06 +0200 (CEST) Date: Thu, 5 Apr 2007 12:50:06 +0200 From: Thomas Kaehn To: Justin Piszcz Cc: Chris Wedgwood , xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS Message-ID: <20070405105005.GF2759@mail3b.westend.com> References: <20070404130535.GE18320@mail3b.westend.com> <20070404154523.GA20096@tuatara.stupidest.org> <20070405072803.GB2759@mail3b.westend.com> <20070405090333.GE2759@mail3b.westend.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.9i X-archive-position: 11055 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: tk@westend.com Precedence: bulk X-list: xfs Hi Justin, On Thu, Apr 05, 2007 at 06:21:43AM -0400, Justin Piszcz wrote: > >So there is no improvement for a single disk. > > > What kind of disks are you using? Maybe just slow disks?? I am using the following disks: http://www.westerndigital.com/en/products/products.asp?DriveID=233 When disabling write barriers delete times are OK. I think that the 3ware RAID controller could have a problem with it. I'll try to contact 3ware in order to come to know if this feature is supported or not. Additionally I am going to try out some advices presented in the 3ware knowledge base. Ciao, Thomas -- Thomas Kähn WESTEND GmbH | Internet-Business-Provider Technik CISCO Systems Partner - Authorized Reseller Im Süsterfeld 6 Tel 0241/701333-18 tk@westend.com D-52072 Aachen Fax 0241/911879 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608 Geschäftsführer: Thomas Neugebauer, Thomas Heller, Michael Kolb From owner-xfs@oss.sgi.com Thu Apr 5 04:11:43 2007 Received: with ECARTIS (v1.0.0; list xfs); Thu, 05 Apr 2007 04:11:46 -0700 (PDT) Received: from lucidpixels.com (lucidpixels.com [75.144.35.66]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id l35BBgfB018902 for ; Thu, 5 Apr 2007 04:11:43 -0700 Received: by lucidpixels.com (Postfix, from userid 1001) id DFC69F063B21; Thu, 5 Apr 2007 07:11:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by lucidpixels.com (Postfix) with ESMTP id D8A5B717672A; Thu, 5 Apr 2007 07:11:41 -0400 (EDT) Date: Thu, 5 Apr 2007 07:11:41 -0400 (EDT) From: Justin Piszcz X-X-Sender: jpiszcz@p34.internal.lan To: Thomas Kaehn cc: Chris Wedgwood , xfs@oss.sgi.com Subject: Re: Strange delete performance using XFS In-Reply-To: <20070405105005.GF2759@mail3b.westend.com> Message-ID: References: <20070404130535.GE18320@mail3b.westend.com> <20070404154523.GA20096@tuatara.stupidest.org> <20070405072803.GB2759@mail3b.westend.com> <20070405090333.GE2759@mail3b.westend.com> <20070405105005.GF2759@mail3b.westend.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1463747160-1637025469-1175771501=:17700" X-archive-position: 11056 X-ecartis-version: Ecartis v1.0.0 Sender: xfs-bounce@oss.sgi.com Errors-to: xfs-bounce@oss.sgi.com X-original-sender: jpiszcz@lucidpixels.com Precedence: bulk X-list: xfs This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1463747160-1637025469-1175771501=:17700 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Thu, 5 Apr 2007, Thomas Kaehn wrote: > Hi Justin, > > On Thu, Apr 05, 2007 at 06:21:43AM -0400, Justin Piszcz wrote: >>> So there is no improvement for a single disk. >>> >> What kind of disks are you using? Maybe just slow disks?? > > I am using the following disks: > > http://www.westerndigital.com/en/products/products.asp?DriveID=3D233 > > When disabling write barriers delete times are OK. I think that > the 3ware RAID controller could have a problem with it. I'll try to > contact 3ware in order to come to know if this feature is supported or > not. > > Additionally I am going to try out some advices presented in the > 3ware knowledge base. > > Ciao, > Thomas > --=20 > Thomas K=E4hn WESTEND GmbH | Internet-Business-Provid= er > Technik CISCO Systems Partner - Authorized Reseller > Im S=FCsterfeld 6 Tel 0241/701333-18 > tk@westend.com D-52072 Aachen Fax 0241/911879 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -