>-Andi Kleen [05 Dec 2001 20:24 +0100] wrote:
>
> On Wed, Dec 05, 2001 at 02:19:27PM -0500, Xianglong Yuan wrote:
> > >-Andi Kleen [05 Dec 2001 19:26 +0100] wrote:
> > >
> > > > Thanks, Steve, it becomes much clear to me now as why it is. This
> > > > problem is critical to us as we run structure simulation for up
> > > > to several weeks and dump intermediate data every couple hours
> > > > and append them to few large files. If for some reason the system
> > > > crash during data output (though rarely), all the results from
> > > > few weeks running will gone which is unacceptable for us.
> > > > Probably I should looking for FS journaling both meta-data and
> > > > file-data, although I don't really need file-data journaling if
> > > > ever I could get my old content back.
> > >
> > > The scenario Steve described obviously does not apply to appending to big
> > > files, because there is no rename() from a temporary file involved.
> > > If it has been flushed they will be on disk. You can force flushing
> > > by using a fsync() for example.
> > >
> > > -Andi
> >
> > But only be on disk after flushing, correct? What if crashing
>
> Yes, but until that the old data is not gone because the file is not
> truncated .
> The problem with the editors is that the old data is usually lost in some
> temporary file; that won't be the problem here.
>
> > during data dumping before it can issue a fsync(), or even during
> > fsync()? I think the best bet is to remove the old file only
>
> Then you lose a few seconds at worst.
>
> > after ensuring the new one is there, or fallback to old one, an
> > asynchronous trasaction which is I believe what Steve said he is
> > working on. Same technique as ordered mode in ext3?
>
> The problem is really that the delete part of rename is synchronous,
> which causes a log flush at an very inconvenient moment if you're using an
> editor
> and crashed shortly afterwards. For appending to existing files it is no
> problem
> though again (first no delete and even if there was a flush it wouldn't
> truncate
> your file)
>
>
> -Andi
>
I see what I am testing using vim is different from what my
simulation actually recording data appendingly :-<< But, anyhow,
ordered mode in ext3 seems to be my best bet. I'll do some
reading to find out the cons in ext3 :-)) Any comments on ext3
will be highly welcomed. Please send me privately if off-topic.
Thanks.
Xianglong
|