I've been meaning to get to this for .. I guess 6 weeks? It's been on
the back burner while I was busy with other stuff, like moving. :)
* Rusty Ballinger (rusty@xxxxxxx) wrote:
> (cc'ing the fam mailing list in case other people may have feedback on adding
> a way to get stat information from fam & a way to ask fam for only the first
> n files when monitoring a large directory.)
Information I would've enjoyed having prior to writing that. :P
> Yes. (It's funny, at every layer from imon -> fam -> application, the amount
> of information which is discarded.) I agree, anything that fam learns from
> stat() should be available to its clients.
Absolutely.
> Here are some random notes on adding this to the API & the protocol
> between fam & its clients:
>
> - I can't break binary compatibility with the existing API (so no
> adding members to the FAMEvent struct, changing arguments to
> FAMMonitorFile(), etc.), but I'm happy to add new structs & functions.
> Unlike the current API & protocol, whatever we add should be flexible
> enough that people can add things in the future without having to come
> up with horrible new function names like FAMOpen4().
You and I had discussed a compability layer for older applications. As
long as we keep that layer in mind, I think we can break it as required.
> - If you want extended events, there ought to be a way to tell fam
> which additional attributes you want. (If you want everything fam can
> get from stat(), you ought to be able to get it; but if all you want
> is one field, you shouldn't have to shuffle the rest of the data
> across the socket.) It seems reasonable to me that these settings be
> per-connection rather than per-request.
Yeah, per session rather than per request seems much wiser to me, and
can be relatively easily addressed.
> - One potential use of the extended information is inside fam itself.
> When fam starts monitoring files/directories which are mounted from a
> remote host, it tries to ask fam on the remote machine to monitor
> those files (much better than polling over NFS). If the remote fam
> can supply all the stat info that the local fam needs, the local fam
> shouldn't have to stat the files. (Neat!)
I have an issue with that only because I don't like that fam must run as
root. While it's certainly nice, it keeps users from being able to use
fam-based applications if they don't have root (for instance, like me at
work). We should find some way to address that problem, without
sacrificing the obvious win in having this (perhaps having fam's
internals view a remote fam or having root like imon: it's nice, but not
required).
Also, does /dev/imon require being accessed by root? Or can an ordinary
user talk to it? If not, something else to consider..
> - Since fam clients might be across the network from fam, the stat
> info has to be portable. (How do we handle a client on a machine with
> a 32-bit off_t famming a large file on a machine with a 64-bit off_t,
> for example?)
I expect someone's dealt with this before, so we'll just have to find
out who and see what they did, and hopefully just follow their
conventions. ;)
> - As the amount of information fam provides to clients goes up, so
> does the risk of a buggy fam being a useful tool for hostile clients.
> I think fam is secure but it's one more thing to lose sleep over if
> you're so inclined.
Security is definately a big deal for any root application. I wonder
what kind of chroot()ing et al (e.g, capabilities under Linux) we could
add in that would make unforseen bugs less dangerous.
> No... & I think most (all?) fs's don't even store them sorted in an
> order which is useful to people; ls -1f on both xfs & ext2 gives you
> the files in whatever order the filesystem found most useful to store
> them in. I think this means that *any* process (even in kernel space)
> has to traverse the entire directory before giving you any portion of
> a sorted list. (And I believe imon doesn't do this, in case you're
> saying it already does!)
Yeah. I still like it though. Sure, you pay an up-front cost, but the
payoff can be very advantagous.
I'm not suggesting imon does this :)
> Also, it seems like when you ask for "the first 15 files", keep in
> mind the kinds of filters & ordering a decent file manager will give
> you: sort by size, by type first & name second, by modification date,
> reverse, only show me *.txt, include hidden files, etc. Anything
> which is going to give you "the first 15 files" ought to be flexible
> enough to let you specify the sort criteria, file globs for filtering,
> and the locale. (Is there some existing library which does this?
> Seems like a generally useful facility.)
I would see this as per-session as well, and I would definately want the
number of sorting algorithms limited, to keep complexity down.
> Now, in the case where fam is looking at a 1000-file directory, a lot
> of traffic can be saved if it can pass back only the "first" 15
> files... so this flexibility, although hard to do, might be worth the
> effort. (On the other hand, what's a client application going to do
> once it's got the first 15 files? Ask for the remaining 985, I
> think... But still, if the application can remain responsive while
> this is happening, and without simply processing n events per visit
> from the main select() loop, that's an improvement.)
I think some applications may immediately ask for the other files, but
newer applications can do that smarter. The payoff is in reaction time:
after the first read, every subsequent access to that directory yields
significantly less traffic. The application would no longer have any
good reason to cache file lists between uses, if it does it, and if it
doesn't, it would take much less time to go from request for files to
display.
Of course, it's just a theory. I'd like to see if it pans out.
--
Source code, list archive, and docs: http://oss.sgi.com/projects/fam/
To unsubscribe: echo unsubscribe fam | mail majordomo@xxxxxxxxxxx
|