Howdy,
I just ran into a problem, that might already be fixed (I'm on pcp 2.1.4) but
I thought I'd bring it up. It concerns pmlogger when you are monitoring lots
of systems.
Basically there is a seperate process that runs for each machine that you are
logging. I'm sure that this makes the gathering of data and such much
quicker, but it does have a drawback when the number of machines you are
monitoring gets up high, like several hundred or thousand.
Basically the problem is this. According to the error message I have, the VFS
(Virtual File Server) running on Linux can only access a maximum of 4096 files
at a time. After that the machine basically goes belly up. So if you have
250 loggers going, each of them normally open 5 files, you have 1250 files
open. Now when you do the log rotate, I can't tell for sure, but I believe
you have a minumum of 10 files open, and possibly 15, the number jumps to 2500
(for 10 files) plus the original 1250 equals 3750, which is getting awfully
close to the limit. If it is 15, your already there.
OK, so you can guess why I'm writting this, yesterday, I added 50 more
machines to my logger, and at log rotation time, the machine choked. (Just to
note, it didn't crash, you just couldn't actually do anything useful)
Anyway, this is a problem that probrubly needs to be looked at.
Troy
--
__________________________________________________
Troy Dawson dawson@xxxxxxxx (630)840-6468
Fermilab ComputingDivision/OSS CSS Group
__________________________________________________
|