Hi Timothy,
thank you very much for your valuable input and your detailed discussion
of xfsrestore's dump tree construction algorithm. I'd prefer to continue
this issue on the list so others can learn from and contribute to, too.
> > "No one cares if you can back up - only if you can recover." (W. Curtis
> > Preston, Unix Backup & Recovery, O'Reilly, 1999, xiv)
> I should have a look at this book.
http://www.oreilly.com/catalog/unixbr/
I think I'll construct some testbeds, i.e. little filesystems with some
hard links in it, do xfsdump, create some more links, do an incremental
dump, create some more links, do an incremental dump of a higher level,
create some more links...
Regards,
Bernie
> > > > xfsrestore.debug.3 told me xfsrestore had deleted user/be/Sent/xy/290.:
> > > >
> > > > [...]
> > > > /usr/sbin/xfsrestore: rename nondir user/be/Sent/xy/290. to
> > > > orphanage/9074670.1
> > > > /usr/sbin/xfsrestore: unlink nondir orphanage/9074670.1
> > > > [...]
> > > > /usr/sbin/xfsrestore: restoring user/be/Sent/xy/291. (9074676 0)
> > > > /usr/sbin/xfsrestore: restoring regular file ino 9074676
> > > > user/be/Sent/xy/291.
> > > > /usr/sbin/xfsrestore: truncating user/be/Sent/xy/291. from 0 to 775
> > > > /usr/sbin/xfsrestore: restore complete: 76 seconds elapsed
> > > >
> > > > What's going on here? I'm using xfsdump-1.0.9-0.
> > >
> > > It looks like that this is happening in the function
> > > xfsdump/restore/tree.c/proc_hardlinks_cb().
> > > I'm not sure what this code is trying to do !
> > > However, I'll post a bug on this at SGI and
> > > look into it further.
> > > Thanks for the report.
> >
> > I just investigated it a little further. I did another cumulative
> > restore of /var/spool/imap (based on level 0 on 2001-07-12). This time
> > I'm using xfsdump-1.0.11-0.
> > (/usr/sbin/amrestore -p $TAPE ente imap | /usr/sbin/xfsrestore -r -v
> > trace - .) > xfsrestore.debug.0
> > (/usr/sbin/amrestore -p $TAPE ente imap | /usr/sbin/xfsrestore -r -v
> > trace - .) > xfsrestore.debug.1
> > (/usr/sbin/amrestore -p $TAPE ente imap | /usr/sbin/xfsrestore -r -v
> > trace - .) > xfsrestore.debug.2
> > (/usr/sbin/amrestore -p $TAPE ente imap | /usr/sbin/xfsrestore -r -v
> > trace - .) > xfsrestore.debug.3
> > (/usr/sbin/amrestore -p $TAPE ente imap | /usr/sbin/xfsrestore -r -v
> > trace - .) > xfsrestore.debug.4
> >
> > One of the files missing after the complete restore is user/ap/in/52.
> > xfsrestore told me it was deleted applying level 2:
> > # grep user/ap/in/52. xfsrestore.debug.?
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: link user/ap/in/MKSports/21. to
> > user/ap/in/52. (4504894 0)
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: linking user/ap/in/MKSports/21.
> > to user/ap/in/52.
> > xfsrestore.debug.2:/usr/sbin/xfsrestore: unlink user/ap/in/52.
> >
> > But it's still there in /var/spool/imap (where the dump was taken from):
> > # ls -li /var/spool/imap/user/ap/in/52.
> > 4504894 -rw------- 2 cyrus root 5464 Apr 13 14:12
> > /var/spool/imap/user/ap/in/52.
> >
> > So it has a hardlink to it. What's the other filename pointing to the
> > inode 4504894?
> > # find /var/spool/imap/user/ap -inum 4504894 -ls
> > 4504894 8 -rw------- 2 cyrus root 5464 Apr 13 14:12
> > /var/spool/imap/user/ap/in/MKSports/21.
> > 4504894 8 -rw------- 2 cyrus root 5464 Apr 13 14:12
> > /var/spool/imap/user/ap/in/52.
> >
> > In the restored tree, there's only user/ap/in/MKSports/21.:
> > # ll user/ap/in/MKSports/21.
> > -rw------- 1 cyrus root 5464 Apr 13 14:12
> > user/ap/in/MKSports/21.
> > No other link to it.
> >
> > This file was restored in level 0, too:
> > # grep user/ap/in/MKSports/21. xfsrestore.debug.?
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: restoring
> > user/ap/in/MKSports/21. (4504894 0)
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: restoring regular file ino
> > 4504894 user/ap/in/MKSports/21.
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: truncating
> > user/ap/in/MKSports/21. from 0 to 5464
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: link user/ap/in/MKSports/21. to
> > user/ap/in/52. (4504894 0)
> > xfsrestore.debug.0:/usr/sbin/xfsrestore: linking user/ap/in/MKSports/21.
> > to user/ap/in/52.
> >
> > Ok, let's grep some more...
> > # grep -n "unlink " /usr/src/linux-2.4-xfs/cmd/xfsdump/restore/*
> > /usr/src/linux-2.4-xfs/cmd/xfsdump/restore/tree.c:1442:
> > "unlink %s\n",
> > (CVS checkout of this morning)
> >
> > This line is in the function noref_elim_recurse (line 1181 in tree.c)
> > and there's a comment after the else in line 1330:
> > /* determine if we can unlink this node.
> > * if its not real, and not refed, simple.
> > * if real and not refed and there is at least
> > * one unreal refed node and no other real
> > * nodes around, must put this one in orphanage
> > * rather than unlinking it.
> > */
> > So, what's an unreal node? And a node not referred to?
> Good questions.
> I wish I had good answers :)
> I didn't write the code (twas written in '95 and the author has gone)
> so I can only go by the code as well.
> And this part of the code I am not familiar with.
>
> Looking at tree.c:
> #define NF_REAL ( 1 << 0 )
> /* set when the corresponding file/dir has been created in
> * the restore destination.
>
> #define NF_REFED ( 1 << 2 )
> /* indicates node is still referenced according to
> incremental/resumed
> * dump. used to detect dirents no longer used. prior to restoring
> * a dump session, this flag is cleared in all nodes. during the
> dirent
> * restoral, it is set. it may also be set during the adjustment
> * for referenced but undumped directories. NOTE: nodes in the
> * orphanage NEVER have this flag set.
>
> And looking where NF_REAL is set (... |= NF_REAL) it seems to be when
> (1) making the directories (mkdirs_recurse()),
> (2) when restoring the file (tree_cb_links()) and
> (3) when processing hardlinks (proc_hardlinks()).
> All 3 things happen _after_ calling noref_elim_recurse().
> So it seems likely to me that at that point the entry is not considered
> real (i.e. not restored yet).
>
> And looking where NF_REFED is set (... |= NF_REFED),
> (1) when adding a directory entry (tree_addent()),
> which exists in the entry tree already,
> then we mark it as referenced.
> ASIDE: A directory entry is expected to appear at least twice.
> As the directory itself and as an entry in the parent directory.
> So the 2nd time it appears it will be marked as referenced.
> If the entry appears first and thus it doesn't have a parent yet,
> it will be placed as a child of the orhpanage directory.
> (2) when adding a non-directory entry (tree_addent()),
> and the entry already exists (same name and parent),
> then we mark it as referenced.
> [I'm not sure how this happens - need to try examples]
> (3) when adjusting for referenced but undumped dirs (tree_adjref()).
> I think this is something to do with marking the entries of a
> directory as referenced if the directory has not been dumped
> and is referenced (i.e. has not changed and still exists).
>
> > Maybe this lost link user/ap/in/52. was not referred to in the level 0
> > dump? Who's doing the mess? xfsdump not referring to or telling
> > xfsrestore of unreal nodes? Or does xfsrestore get things wrong?
> I would say the problem lies in xfsrestore.
> Xfsrestore is the one which builds a directory-entry tree from
> the initial dump of directory entries. The nodes in this tree are
> what have the NF_* flags set.
>
> Getting back to the problem....
> I'll have to continue this later, I need to get back to some
> IRIX bugs.
>
> Later,
> Tim.
|