xfs
[Top] [All Lists]

Re: tuning, many small files, small blocksize

To: Linux XFS <xfs@xxxxxxxxxxx>
Subject: Re: tuning, many small files, small blocksize
From: pg_xfs2@xxxxxxxxxxxxxxxxxxx (Peter Grandi)
Date: Tue, 19 Feb 2008 21:24:34 +0000
In-reply-to: <47BAC139.3000109@xxxxxxxxxxxxxxxxxxxx>
References: <e03b90ae0802152101t2bfa4644kcca5d6329239f9ff@xxxxxxxxxxxxxx> <47BA10EC.3090004@xxxxxxxxx> <20080218235103.GW155407@xxxxxxx> <47BA2AFD.2060409@xxxxxxxxx> <20080219024924.GB155407@xxxxxxx> <e03b90ae0802182058h7a1535c6w749eb46cbe434ef2@xxxxxxxxxxxxxx> <18362.37642.577718.529415@xxxxxxxxxxxxxxxxxx> <47BAC139.3000109@xxxxxxxxxxxxxxxxxxxx>
Sender: xfs-bounce@xxxxxxxxxxx
>>> On Tue, 19 Feb 2008 12:44:57 +0100, Hannes Dorbath
>>> <light@xxxxxxxxxxxxxxxxxxxx> said:

[ ... a collection of millions of small records ... ]

>> That sounds like a good use for a LDAP database, but using
>> Berkeley DB directly may be best. One could also do a FUSE
>> module or a special purpose NFS server that presents a
>> Berkeley DB as a filesystem, but then we would be getting
>> rather close to ReiserFS.

light> During testing of HA clusters some time ago I found BDB
light> to always be the first thing to break. It seems to have
light> very poor recovery and seems not fine with neither file
light> systems snapshots nor power failures. [ ... ]

Sometimes BDB had problems, but that seems in the past. It also
relies critically on some precise behaviour from the storage
layer, filesystem downwards:

  
http://WWW.Oracle.com/technology/documentation/berkeley-db/db/ref/transapp/reclimit.html

If all those conditions are not met, then it cannot do recovery.
Fortunately XFS can meet those conditions (I think also the page
size one), if properly configured and if the hardware does not
lie.

light> Personally I ended up doing this for OpenLDAP and never looked back: 
light> http://www.samse.fr/GPL/ldap_pg/HOWTO/x12.html

Well, PostgresQL is of course a much nicer, more scalable DBMS
than BDB. But for a relatively small, mostly-ro collection of
small records the latter may be appropriate. XFS works with it
fairly well too.


<Prev in Thread] Current Thread [Next in Thread>