(cc'ing the fam mailing list in case other people may have feedback on adding
a way to get stat information from fam & a way to ask fam for only the first
n files when monitoring a large directory.)
> Raster told me he didn't see a ton of use of the ability to sort, but he
> like to be able to store metadata.. of course, efm style. I told him that'd
> be bad, so I suggested arbitrary metadata per file. We also agreed fam
> should be capable of providing file permissions, since it's already
> stat()ing the file on behalf of the process, as well as providing any other
> useful information that can be obtained from stat().
Yes. (It's funny, at every layer from imon -> fam -> application, the amount
of information which is discarded.) I agree, anything that fam learns from
stat() should be available to its clients. Here are some random notes on
adding this to the API & the protocol between fam & its clients:
- I can't break binary compatibility with the existing API (so no adding
members to the FAMEvent struct, changing arguments to FAMMonitorFile(),
etc.), but I'm happy to add new structs & functions. Unlike the current
API & protocol, whatever we add should be flexible enough that people
can add things in the future without having to come up with horrible
new function names like FAMOpen4().
- If you want extended events, there ought to be a way to tell fam which
additional attributes you want. (If you want everything fam can get from
stat(), you ought to be able to get it; but if all you want is one field,
you shouldn't have to shuffle the rest of the data across the socket.)
It seems reasonable to me that these settings be per-connection rather
than per-request.
- One potential use of the extended information is inside fam itself.
When fam starts monitoring files/directories which are mounted from a
remote host, it tries to ask fam on the remote machine to monitor those
files (much better than polling over NFS). If the remote fam can supply
all the stat info that the local fam needs, the local fam shouldn't have
to stat the files. (Neat!)
- Since fam clients might be across the network from fam, the stat info
has to be portable. (How do we handle a client on a machine with a 32-bit
off_t famming a large file on a machine with a 64-bit off_t, for example?)
- As the amount of information fam provides to clients goes up, so does
the risk of a buggy fam being a useful tool for hostile clients. I think
fam is secure but it's one more thing to lose sleep over if you're so
inclined.
> His main reason against fam sorting the directory is that it couldn't
> respond to the process until it read the whole directory. That's legitimate
> IMHO, but I still want to find a way to do it so fam can return just the
> first 15 files. Know of a way to get the fs to return it sorted, without
> requiring imon? Raster suggested that you query fam at run-time for extra
> features only provided by imon, i.e., the ability to get the first 15 files
> alphabetically-sorted.
No... & I think most (all?) fs's don't even store them sorted in an order
which is useful to people; ls -1f on both xfs & ext2 gives you the files in
whatever order the filesystem found most useful to store them in. I think
this means that *any* process (even in kernel space) has to traverse the
entire directory before giving you any portion of a sorted list. (And I
believe imon doesn't do this, in case you're saying it already does!)
Also, it seems like when you ask for "the first 15 files", keep in mind the
kinds of filters & ordering a decent file manager will give you: sort by
size, by type first & name second, by modification date, reverse, only show
me *.txt, include hidden files, etc. Anything which is going to give you
"the first 15 files" ought to be flexible enough to let you specify the sort
criteria, file globs for filtering, and the locale. (Is there some existing
library which does this? Seems like a generally useful facility.)
Now, in the case where fam is looking at a 1000-file directory, a lot of
traffic can be saved if it can pass back only the "first" 15 files... so
this flexibility, although hard to do, might be worth the effort. (On the
other hand, what's a client application going to do once it's got the first
15 files? Ask for the remaining 985, I think... But still, if the
application can remain responsive while this is happening, and without
simply processing n events per visit from the main select() loop, that's an
improvement.)
--Rusty
--
Source code, list archive, and docs: http://oss.sgi.com/projects/fam/
To unsubscribe: echo unsubscribe fam | mail majordomo@xxxxxxxxxxx
|