xfs
[Top] [All Lists]

NOW: o_direct -- WAS: Re: WARNING in xfs_lwr.c, xfs_write()

To: xfs@xxxxxxxxxxx
Subject: NOW: o_direct -- WAS: Re: WARNING in xfs_lwr.c, xfs_write()
From: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Wed, 26 May 2010 10:07:18 -0500
In-reply-to: <20100526070620.GT2150@dastard>
References: <20100523002023.41f5a5c8@xxxxxxxxxxxxxxxxxxxxxxx> <20100523101856.GL2150@dastard> <20100523092344.0fcaab42@xxxxxxxxxxxxxxxxxxxxxxx> <4BF9FCA8.8090906@xxxxxxxxxxxxxxxxx> <20100524143428.6f3a117c@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <20100526070620.GT2150@dastard>
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
Dave Chinner put forth on 5/26/2010 2:06 AM:
> On Mon, May 24, 2010 at 02:34:28PM -0500, Roman Kononov wrote:
>> On Sun, 23 May 2010 23:12:24 -0500 Stan Hoeppner
>> <stan@xxxxxxxxxxxxxxxxx> wrote:
>>> "The whole notion of "direct IO" is totally braindamaged. Just say no.
>> ...
>>> From:  http://lkml.org/lkml/2007/1/10/235
>>
>> I definitely measure dramatic overall performance benefit using
>> O_DIRECT carefully.
>>
>> In that thread, it is doubtful that madvise+mmap+msync allow
>> asynchronous zero-copy reads and writes to/from already pinned by a
>> device driver memory of data produced/consumed by that device, without
>> cache pollution and with intelligent handling of disk errors. Am I
>> wrong?
> 
> No, you are not wrong.
> 
> Remember, just because Linus asserts something it doesn't mean he is
> right. Yes, he's right an awful lot of the time, but not always.  In
> this case, most people with experience in writing high performance
> IO engines with tell your that mmap() and advisory interfaces are no
> substitute for the fine grained control of IO issue that direct IO
> provides you with.
> 
> And in the case of XFS, mmap serialiseѕ write page faults to
> different areas of the same file, whereas direct IO allows
> concurrent reads and writes to different regions of the same file.
> That makes direct IO far more scalable than than any mmap interface
> will ever be....
> 
> Cheers,
> 
> Dave.

Please educate the ignorant a little bit Dave.  I'm not a programmer, or at
least, haven't been one for a couple of decades.  If o_direct is superior to
mmap, why then don't, say, Postfix and Dovecot use it instead of mmap?  Email
servers are some of the most disk I/O bound applications on the planet.  I
would think on heavily loaded mail servers (smtp or imap), at $big_isp for
example, buffer cache would yield very little performance gain, and may even
slow the system down due to buffer cache thrashing.

Why do you think Wietse and Timo don't use o_direct instead of mmap?  Timo is
working on a complex and aggressive totally asynchronous I/O subsystem for a
future Dovecot release in an effort to speed up I/O on loaded systems.  Could
o_direct not be the solution?  AFAIK, both Postfix and Dovecot support running
on just about every Unix like OS on the planet.  Is o_direct not a portable
interface, limited to Linux only?  Is o_direct a POSIX standard?

I'm sure o_direct isn't the best fit for many I/O bound applications. I'm just
trying to understand why.  What applications are the best candidates for using
o_direct?  Why would one want to avoid using o_direct in an I/O bound 
application?

My apologies for the newbish questions.  If this has all been asked/answered
before, please just point me to any papers that explain the pros/cons of
o_direct.  Thanks.

-- 
Stan

<Prev in Thread] Current Thread [Next in Thread>