These notes on imon consist of the following sections: 0. What's in this patch 1. How imon works 2. How imon should work 3. The relationship between fam & imon 4. Open/unresolved imon issues 5. imon design flames Sections 1, 2, and 4 are probably the most interesting. Send feedback to fam@oss.sgi.com or rusty@sgi.com. imon was originally built by Wiltse Carpenter and/or Bruce Karsh on IRIX. It was ported to Linux by Roger Chickering. A few additional bits, such as portions of these notes, may have been done by Rusty Ballinger, but good luck getting that guy to admit to anything. 0. What's in this patch This patch is for 2.4.0-test9 kernels. It is EXPERIMENTAL, ONLY SLIGHTLY TESTED, and portions of it should be rewritten; see section 2, "How imon should work." Modified files: Documentation/Configure.help added documentation for CONFIG_IMON fs/Config.in added "source fs/imon/Config.in" fs/Makefile added chunk for building imon fs/attr.c added IMON_EVENT in notify_change, which should be removed fs/exec.c added CONFIG_EXECOUNT & IMON_EVENT in do_execve; I'm not sure whether this can be removed fs/filesystems.c added init_imon(), which is only used if imon is compiled in rather than being a module? Can this be removed? imon should probably always be a module. fs/namei.c added IMON_EVENT in various functions, all of which should be removed fs/read_write.c added IMON_EVENT in sys_write and sys_writev, which should be removed fs/super.c (no change - should have added unmount notification?) include/linux/fs.h added unsigned int i_execount to inode struct include/linux/sched.h added struct dentry *script to task_struct kernel/exit.c added CONFIG_EXECOUNT stuff in do_exit kernel/fork.c added CONFIG_EXECOUNT stuff in do_fork kernel/ksyms.c added global imon symbols, which should be removed as soon as possible New files: Documentation/imon.txt fs/imon/Config.in fs/imon/Makefile fs/imon/imon.c fs/imon/imon_static.c include/linux/imon.h 1. How imon works When a process opens /dev/imon and uses the IMONIOC_EXPRESS ioctl to express interest in directories & files, imon sticks those interests in a hash table. Various file operations conclude with a call to IMON_EVENT() or IMON_BROADCAST, which are macros which call imon_event() or imon_broadcast() if the global flag imon_enabled is set. Inside of imon_event, if the dev/inode on which the event occurred is in the hash table of expressed interests, the event is put in the event queue, which the client process can read() from /dev/imon. When imon has no clients, the imon_enabled flag is cleared, and file operations do not involve imon. (In a kernel configured without imon support, the IMON_EVENT and IMON_BROADCAST macros are no-ops.) The imon changes also include support for tracking how many instances of an executable are currently running. This is done by adding to the inode struct a count of the number of processes executing it. Roger's notes: When exec is called, the inode corresponding to the file being exec'd has its i_execount field incremented, while the i_execount for the previous inode being executed is decremented. The inode is kept track of in the per-task (process) field called script (which is actually a struct dentry which contains the inode of interest). The reason the field is called script is because it can be used to keep track of exec'd shell scripts. The exec code in the kernel actually sets up the process as if the shell were run instead of the shell script, and the script/i_execount logic is used to remember the fact that the shell script was run so that can be represented to the user. I originally tried to implement IMON_EXEC/IMON_EXIT by looking at memory regions mapped with execute permission, but this did not work for shell scripts. Bookkeeping for the i_execount field is done in three places: (1) In exec, as described above (2) In fork, i_execount must be incremented, because the child process is still executing the same inode (3) In exit, i_execount is decremented. exec and exit notify imon when the flag gains or loses a value of 0. The i_execount field of struct inode and the script field of struct task_struct are not #ifdef CONFIG'd out because I didn't want to introduce problems with different .o files having different notions of the sizes of these very important and fundamental data structures. 2. How imon should work When imon beings monitoring an inode, it should replace that inode's table of file & inode operations with its own tables of functions that call the original functions and then post the appropriate event on the queue. This way, the only file operations intercepted by imon are on files that imon is monitoring. As it is now, as long as imon is monitoring at least one file, operations on *any* file cause imon to check its hash table to determine whether the operation was on a monitored file. This is not so good both because of the unnecessary work being performed, and because of the IMON_EVENT() macro calls which must be scattered throughout the filesystem code. (If this is changed, then hopefully there won't be any imon code anywhere but the imon module.) It was done this way for a couple of reasons. A big one is that Linux has a lot of "weird" filesystems which might perform their own alteration of the inode operations, and there wasn't a good way to keep them & imon from stomping each other. It would be nice if someone who's familiar with the filesystem code would spend an evening or two and fix it. (I think it would take me longer than that, and the quality of the results would be questionable.) 3. The relationship between fam & imon fam, the File Alteration Monitor, is a user-level daemon; it's the way client applications talk to imon. All client requests go to fam; fam determines which requests to pass on to imon, reads events from /dev/imon, and passes them on to the appropriate clients. On top of the local inode monitoring performed by imon, fam provides the following services: - monitoring remote files - serving remote requests for local files - handling interest relocation when filesystems are unmounted - monitoring deleted files - monitoring local files when the kernel isn't configured for imon support When fam is running on a system which doesn't have kernel support for imon, it polls files. This means that an application which uses the fam API will work even on systems which do not have imon support enabled in the kernel. (The difference is that there will be more latency between file operations & event notification. Also, EXEC and EXIT events will not be delivered, but most applications don't require this.) 4. Open/unresolved imon issues All of these issues are trivia compared to the way imon hooks into the filesystem. (See above.) - If the IMON_ATTR event type is not needed, it should be removed from imon.h. It may be needed for famming individual files (or getting notification about a fammed directory getting deleted). - Unmount notification has not been tested, and therefore probably does not work. XXX YOU NEED TO VERIFY whether a kdev_t can be treated as a dev_t. - [This may no longer be true; it was happening on 2.2.13.] On my system, every second I get a bunch of seemingly-bad events on device 0 with the i_sb or i_sb->s_type == NULL. (Is this /proc? Should those events be discarded?) According to kdev_t.h, device 0 is "no device," so perhaps I should discard them without checking the hash table. - I think the reorganize-collision has a logic error which may affect performance. Here are my notes, which may no longer be correct: 0 1 2 3 Start Insert 3a 3a Insert 3b 3b 3a* Insert 2a 2a 3b* 3a* Insert 3c 3c 2a* 3b* 3a* Delete 3b 3c 2a* 3a* Delete 3a 2a* 3c At this point, you have 2a incorrectly marked as a collision. What is the harm in this? It could result in slightly slower probes in some cases? Is that all? 0 1 2 3 Start Insert 3a 3a Insert 3b 3b 3a* Insert 1a 1a 3b 3a* At this point, if you delete 3a, I think you will traverse more of the hash table than you need to, trying to reorganize elements which are already correctly hashed. How bad is this? 5. imon design flames Sorry this section isn't as interesting as its title implies. I was going to add some of the exchange here about whether imon should be an exclusive driver (as it is), or whether it should allow any number of clients to open /dev/imon at once. I did actually have a version which worked that way (it was pretty neat), but when I told other people about it, they presented some compelling arguments about why it was better to have it implemented the way it was. I may add that argument here, as it will save people some time in the future if they want to repeat it.