Re: DMAPI questions...

[Dean Roehrich <roehrich@sgi.com>]

>From:  Josh Helmer <joshhelmer@cox.net>
>Hey folks...
>
>Perhaps this is not the appropriate place to ask this... If not, please 
>forgive me, but this looks like just about the best place to get the feedback 
>I am looking for.
>
>I am in the early design stages of some backup/HSM software.   At this point I
> 
>am just researching my options and I was hoping someone could help me out 
>with some questions I have about DMAPI in general (since XFS seems to be the 
>only place where anyone is working in this area under linux  - at least since 
>the OpenXDSM project died) and about the XFS implementation of DMAPI.
>
>One of our requirements is near-real-time backup.  Basically what we need to 
>be able to do is duplicate changes to secondary storage as soon as possible, 
>and then after a file has aged we will go back and poke holes as necessary.   
>When a file is accessed we would fill in the holes again.   Unfortunately our 
>secondary storage is SIGNIFICANTLY slower than a hard drive (writes at 
>between 1.0 and 1.5Mb/s).  Rather than trying to duplicate every write, we 
>were thinking about just adding files to a backup queue when the file is 
>closed.  Unfortunately, the easist way that I can see for handling this is to 
>handle the debut event (initialize the event list for a file) and the close 
>event(schedule the file for backup).  We understand that this method will not 
>ensure EVERY change to a file is backed up, but for our application, files 
>will not change often enough to make this a big concern.
>
>Of course, this will only work if XFS adds support for the debut and close 
>events.  Are there any plans to implement these events in XFS?  If so, is 
>there an estimate on when this will be available?  If not, is there any other 
>way of handling this that I have not considered yet.


DEBUT is for use on filesystems that don't allow some form of persistent
dmattrs.  We have persistent dmattrs.  We're not going to implement this event
ourselves.

CLOSE has never been on our to-do list.


>I have also considered creating a separate write queue.  In this 
>implementation I would basically try to consolidate writes (a non-trivial 
>operation with questionable advantages - may just forget about trying that) 
>and then after no write events have appeared after a pre-determined amount of 
>time, flush all writes to the secondary storage.  As I stated earlier I am 
>still in the VERY early stages of design and I have probably overlooked some 
>issues (locking and file consistency... ugh...  LOTS of work to do there...) 
>with this approach too.  Anyhow, any input would be appreciated.
>
>Maybe I would be better off just going with the standard NFS-server solution?


Have you spent any time researching existing HSMs?  Can we speak in terms of
that experience?

You speak about wanting near-realtime backup, but say you can't write to the
backup device fast enough to do this--which means you're trying for something
else.  So do you really need to have it appear on the backup device at the
same time it appears on the primary device, or do you just need to make sure
that the primary device always appears "bottomless" and that writes to it
never block?

Dean