From simonf@cshl.edu Tue Jan 4 13:56:52 2005 Received: with ECARTIS (v1.0.0; list fam); Tue, 04 Jan 2005 13:57:00 -0800 (PST) Received: from cshl.edu (phage.cshl.edu [143.48.1.1]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j04LuWkt008791 for ; Tue, 4 Jan 2005 13:56:52 -0800 Received: from [143.48.3.70] (shaman.cshl.edu [143.48.3.70]) by cshl.edu (8.11.7+Sun/8.9.1) with ESMTP id j04M51D08338; Tue, 4 Jan 2005 17:05:01 -0500 (EST) Message-ID: <41DB130D.5040608@cshl.edu> Date: Tue, 04 Jan 2005 17:05:01 -0500 From: "Vsevolod (Simon) Ilyushchenko" User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041020 X-Accept-Language: en-us, en MIME-Version: 1.0 To: fam@oss.sgi.com Subject: Problem with a large number of directories. Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.80/645/Mon Dec 27 14:56:20 2004 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 291 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: simonf@cshl.edu Precedence: bulk X-list: fam Hi, Has anyone has used fam with thousands of directories? I have about 6k of them, and halfway through traversing the directory tree the Perl script that I am using hangs up. I am aware that I have to keep reading messages while I am adding directories to avoid filling the message queue, so that is not an issue. I would appreciate any hints on where to look for the problem. Simon -- Simon (Vsevolod ILyushchenko) simonf@cshl.edu http://www.simonf.com Terrorism is a tactic and so to declare war on terrorism is equivalent to Roosevelt's declaring war on blitzkrieg. Zbigniew Brzezinski, U.S. national security advisor, 1977-81 From frank@hemer.org Tue Jan 4 14:13:17 2005 Received: with ECARTIS (v1.0.0; list fam); Tue, 04 Jan 2005 14:13:24 -0800 (PST) Received: from mailout10.sul.t-online.com (mailout10.sul.t-online.com [194.25.134.21]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j04MCqnf009457 for ; Tue, 4 Jan 2005 14:13:17 -0800 Received: from fwd11.aul.t-online.de by mailout10.sul.t-online.com with smtp id 1Clx3L-0003dn-04; Tue, 04 Jan 2005 23:21:23 +0100 Received: from server1.home.de (bjgUjyZdgeLGTm88OhJKPESNXZc39OVmb+jt9E9f-gEGOovHaTtQE9@[84.135.143.193]) by fmrl11.sul.t-online.com with esmtp id 1Clx3E-1H4ey80; Tue, 4 Jan 2005 23:21:16 +0100 Received: from master.home.de (master.home.de [192.168.0.30]) by server1.home.de (8.11.6/8.11.6/SuSE Linux 0.5) with ESMTP id j04MLF801746 for ; Tue, 4 Jan 2005 23:21:15 +0100 From: Frank Hemer Reply-To: frank@hemer.org To: fam@oss.sgi.com Subject: Re: Problem with a large number of directories. Date: Tue, 4 Jan 2005 23:21:15 +0100 User-Agent: KMail/1.5.1 References: <41DB130D.5040608@cshl.edu> In-Reply-To: <41DB130D.5040608@cshl.edu> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200501042321.15292.frank@hemer.org> X-ID: bjgUjyZdgeLGTm88OhJKPESNXZc39OVmb+jt9E9f-gEGOovHaTtQE9@t-dialin.net X-TOI-MSGID: 36969f97-38d7-4974-8d6a-8e3c2cddb99e X-Virus-Scanned: ClamAV 0.80/645/Mon Dec 27 14:56:20 2004 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 292 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: frank@hemer.org Precedence: bulk X-list: fam On Tuesday 04 January 2005 23:05, you wrote: > Hi, > > Has anyone has used fam with thousands of directories? I have about 6k > of them, and halfway through traversing the directory tree the Perl > script that I am using hangs up. > > I am aware that I have to keep reading messages while I am adding > directories to avoid filling the message queue, so that is not an issue. > > I would appreciate any hints on where to look for the problem. Maybe your fd's are exhausted? The linux default as none-root is about 1024? Frank From simonf@cshl.edu Tue Jan 4 15:17:10 2005 Received: with ECARTIS (v1.0.0; list fam); Tue, 04 Jan 2005 15:17:17 -0800 (PST) Received: from cshl.edu (phage.cshl.edu [143.48.1.1]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j04NGnPH012757 for ; Tue, 4 Jan 2005 15:17:10 -0800 Received: from [143.48.3.70] (shaman.cshl.edu [143.48.3.70]) by cshl.edu (8.11.7+Sun/8.9.1) with ESMTP id j04NPGD13096; Tue, 4 Jan 2005 18:25:16 -0500 (EST) Message-ID: <41DB25DB.1040804@cshl.edu> Date: Tue, 04 Jan 2005 18:25:15 -0500 From: "Vsevolod (Simon) Ilyushchenko" User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041020 X-Accept-Language: en-us, en MIME-Version: 1.0 To: frank@hemer.org CC: fam@oss.sgi.com Subject: Re: Problem with a large number of directories. References: <41DB130D.5040608@cshl.edu> <200501042321.15292.frank@hemer.org> In-Reply-To: <200501042321.15292.frank@hemer.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.80/645/Mon Dec 27 14:56:20 2004 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 293 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: simonf@cshl.edu Precedence: bulk X-list: fam Frank, I wish it were the matter of fd's. I'm running as root, and my /proc/sys/fs/file-max is 262k. Thanks, Simon Frank Hemer wrote: > On Tuesday 04 January 2005 23:05, you wrote: > >>Hi, >> >>Has anyone has used fam with thousands of directories? I have about 6k >>of them, and halfway through traversing the directory tree the Perl >>script that I am using hangs up. >> >>I am aware that I have to keep reading messages while I am adding >>directories to avoid filling the message queue, so that is not an issue. >> >>I would appreciate any hints on where to look for the problem. > > > Maybe your fd's are exhausted? The linux default as none-root is about 1024? -- Simon (Vsevolod ILyushchenko) simonf@cshl.edu http://www.simonf.com Terrorism is a tactic and so to declare war on terrorism is equivalent to Roosevelt's declaring war on blitzkrieg. Zbigniew Brzezinski, U.S. national security advisor, 1977-81 From simonf@cshl.edu Tue Jan 4 22:15:25 2005 Received: with ECARTIS (v1.0.0; list fam); Tue, 04 Jan 2005 22:15:34 -0800 (PST) Received: from cshl.edu (phage.cshl.edu [143.48.1.1]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j056F22J005355 for ; Tue, 4 Jan 2005 22:15:25 -0800 Received: from [143.48.3.70] (shaman.cshl.edu [143.48.3.70]) by cshl.edu (8.11.7+Sun/8.9.1) with ESMTP id j05IEoD08064; Wed, 5 Jan 2005 13:14:50 -0500 (EST) Message-ID: <41DC2E9A.1000105@cshl.edu> Date: Wed, 05 Jan 2005 13:14:50 -0500 From: "Vsevolod (Simon) Ilyushchenko" User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041020 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Vsevolod (Simon) Ilyushchenko" CC: fam@oss.sgi.com Subject: Re: Problem with a large number of directories. References: <41DB130D.5040608@cshl.edu> In-Reply-To: <41DB130D.5040608@cshl.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.80/650/Sun Jan 2 19:00:02 2005 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 294 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: simonf@cshl.edu Precedence: bulk X-list: fam To followup on this - I've run the Perl script again, and this time it went through all the directories. Very strange. I've found this in the FAM FAQ: *** Simultaneous requests: Each client process can have up to 1024 active requests. This is a limitation of the select() system call. *** Does anybody know if it's still true and what exactly it means? That I can't request to monitor more than 1024 directories? But it looks like I can, for now. Also, is anybody actively maintaining FAM? In Fedora 3, the FAM package has been dropped in favor of the Gamin project: http://www.gnome.org/~veillard/gamin/index.html Thanks, Simon Vsevolod (Simon) Ilyushchenko wrote: > Hi, > > Has anyone has used fam with thousands of directories? I have about 6k > of them, and halfway through traversing the directory tree the Perl > script that I am using hangs up. > > I am aware that I have to keep reading messages while I am adding > directories to avoid filling the message queue, so that is not an issue. > > I would appreciate any hints on where to look for the problem. > > Simon -- Simon (Vsevolod ILyushchenko) simonf@cshl.edu http://www.simonf.com Terrorism is a tactic and so to declare war on terrorism is equivalent to Roosevelt's declaring war on blitzkrieg. Zbigniew Brzezinski, U.S. national security advisor, 1977-81 From mraymond@sgi.com Tue Jan 4 23:51:30 2005 Received: with ECARTIS (v1.0.0; list fam); Tue, 04 Jan 2005 23:51:37 -0800 (PST) Received: from omx2.sgi.com (omx2-ext.sgi.com [192.48.171.19]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j057pAHR008708 for ; Tue, 4 Jan 2005 23:51:30 -0800 Received: from flecktone.americas.sgi.com (flecktone.americas.sgi.com [198.149.16.15]) by omx2.sgi.com (8.12.11/8.12.9/linux-outbound_gateway-1.1) with ESMTP id j05LFxeS005236 for ; Wed, 5 Jan 2005 13:15:59 -0800 Received: from daisy-e236.americas.sgi.com (daisy-e236.americas.sgi.com [128.162.236.214]) by flecktone.americas.sgi.com (8.12.9/8.12.10/SGI_generic_relay-1.2) with ESMTP id j05Jo2CK5827237; Wed, 5 Jan 2005 13:50:02 -0600 (CST) Received: from goliath.americas.sgi.com (goliath.americas.sgi.com [128.162.232.85]) by daisy-e236.americas.sgi.com (8.12.9/SGI-server-1.8) with ESMTP id j05Jo2q528769853; Wed, 5 Jan 2005 13:50:02 -0600 (CST) Received: from goliath.americas.sgi.com by goliath.americas.sgi.com (SGI-8.12.5/SGI-client-1.7) via ESMTP id j05Jo2Ef022933; Wed, 5 Jan 2005 13:50:02 -0600 (CST) Received: (from mraymond@localhost) by goliath.americas.sgi.com (SGI-8.12.5/8.12.5/Submit) id j05Jo2cM022975; Wed, 5 Jan 2005 13:50:02 -0600 (CST) Date: Wed, 5 Jan 2005 13:50:01 -0600 From: Michael Raymond To: "Vsevolod (Simon) Ilyushchenko" Cc: fam@oss.sgi.com Subject: Re: Problem with a large number of directories. Message-ID: <20050105135001.E21587@goliath.americas.sgi.com> References: <41DB130D.5040608@cshl.edu> <41DC2E9A.1000105@cshl.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <41DC2E9A.1000105@cshl.edu>; from simonf@cshl.edu on Wed, Jan 05, 2005 at 01:14:50PM -0500 X-Virus-Scanned: ClamAV 0.80/650/Sun Jan 2 19:00:02 2005 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 295 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: mraymond@sgi.com Precedence: bulk X-list: fam We'll take patches against FAM if they're supplied, but everyone here is otherwise too busy with other things to actively work on FAM. Michael On Wed, Jan 05, 2005 at 01:14:50PM -0500, Vsevolod (Simon) Ilyushchenko wrote: > To followup on this - I've run the Perl script again, and this time it > went through all the directories. Very strange. > > I've found this in the FAM FAQ: > > *** > Simultaneous requests: Each client process can have up to 1024 active > requests. This is a limitation of the select() system call. > *** > > Does anybody know if it's still true and what exactly it means? That I > can't request to monitor more than 1024 directories? But it looks like I > can, for now. > > Also, is anybody actively maintaining FAM? In Fedora 3, the FAM package > has been dropped in favor of the Gamin project: > > http://www.gnome.org/~veillard/gamin/index.html > > Thanks, > Simon > > Vsevolod (Simon) Ilyushchenko wrote: > > Hi, > > > > Has anyone has used fam with thousands of directories? I have about 6k > > of them, and halfway through traversing the directory tree the Perl > > script that I am using hangs up. > > > > I am aware that I have to keep reading messages while I am adding > > directories to avoid filling the message queue, so that is not an issue. > > > > I would appreciate any hints on where to look for the problem. > > > > Simon > > -- > > Simon (Vsevolod ILyushchenko) simonf@cshl.edu > http://www.simonf.com > > Terrorism is a tactic and so to declare war on terrorism > is equivalent to Roosevelt's declaring war on blitzkrieg. > > Zbigniew Brzezinski, U.S. national security advisor, 1977-81 -- Michael A. Raymond Office: (651) 683-3434 Core OS Group Real-Time System Software From esr@thyrsus.com Fri Jan 14 00:25:45 2005 Received: with ECARTIS (v1.0.0; list fam); Fri, 14 Jan 2005 00:25:54 -0800 (PST) Received: from snark.thyrsus.com (dsl092-053-140.phl1.dsl.speakeasy.net [66.92.53.140]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j0E8Pg6T028334 for ; Fri, 14 Jan 2005 00:25:45 -0800 Received: from snark.thyrsus.com (localhost [127.0.0.1]) by snark.thyrsus.com (8.13.2/8.13.2) with ESMTP id j0E8NiAu003886 for ; Fri, 14 Jan 2005 03:23:44 -0500 Date: Fri, 14 Jan 2005 03:23:44 -0500 From: esr@thyrsus.com Message-Id: <200501140823.j0E8NiAu003886@snark.thyrsus.com> To: fam@oss.sgi.com Subject: problems in fam.3x X-Virus-Scanned: ClamAV 0.80/650/Sun Jan 2 19:00:02 2005 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Status: Clean X-archive-position: 296 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: esr@thyrsus.com Precedence: bulk X-list: fam This is automatically generated email about problems in a man page for which you appear to be responsible. If you are not the right person or list, tell me and I will attempt to correct my database. See http://catb.org/~esr/doclifter/problems.html for details on how and why these patches were generated. Feel free to email me with any questions. Note: These patches do not change the mod date of any manual page. You may wish to do that by hand. Problems with fam.3x: 1. Unknown or invalid macro. That is, one that does not fit in the macro set that the man page seems to be using. This is a serious error; it often means part of your text is being lost or rendered incorrectly. 2. Broken command synopsis syntax. This may mean you're using a construction in the command synopsis other than the standard [ ] | { }, or it may mean you have running text in the command synopsis section (the latter is not technically an error, but it's impossible to translate into DocBook markup). --- fam.3x-orig 2003-11-30 04:55:50.000000000 -0500 +++ fam.3x 2003-11-30 04:56:36.000000000 -0500 @@ -1,5 +1,5 @@ '\"macro stdmacro -.if n .pH g3x.fam @(#)fam 30.3 of 1/19/86 +.\" g3x.fam @(#)fam 30.3 of 1/19/86 .nr X .if \nX=0 .ds x} FAM 3X "Specialized Libraries" "\&" .if \nX=1 .ds x} FAM 3X "Specialized Libraries" @@ -79,8 +79,6 @@ .SH USING FAM Here are the steps required to use \fIFAM \fPin an application: .PP -.AL -.LI .IP 1. Create a connection to \fIfam\fP by calling FAMOpen. This routine will pass back a FAMConnection structure used in all \fIfam \fP @@ -262,8 +260,6 @@ .PP There are two ways to for applications to receive \fIFAM \fPevents: .PP -.AL -.LI 1. The Select approach - The application selects on the file descriptor returned from FAMOpen, in the FAMConnection structure. When this file descriptor becomes active, the application calls ----------------------------- -- Eric S. Raymond From jancs@dv.lv Sun Jan 16 08:58:49 2005 Received: with ECARTIS (v1.0.0; list fam); Sun, 16 Jan 2005 08:58:54 -0800 (PST) Received: from mephisto.ktf.rtu.lv ([213.175.91.1]) by oss.sgi.com (8.13.0/8.13.0) with ESMTP id j0GGwlGA019004 for ; Sun, 16 Jan 2005 08:58:48 -0800 Received: from diablo.integoplus.lv (diablo.integoplus.lv [213.21.215.5]) by mephisto.ktf.rtu.lv (8.12.11/8.12.11) with ESMTP id j0GGwdax014932 for ; Sun, 16 Jan 2005 18:58:39 +0200 Received: by diablo.integoplus.lv (Postfix, from userid 99) id 9C49634A9D; Sun, 16 Jan 2005 18:58:39 +0200 (EET) Received: from ns.4nets.lv (ns.4nets.lv [217.199.126.2]) by inbox.dv.lv (IMP) with HTTP for ; Sun, 16 Jan 2005 18:58:39 +0200 Message-ID: <1105894719.41ea9d3f691d2@inbox.dv.lv> Date: Sun, 16 Jan 2005 18:58:39 +0200 From: Jancs To: fam@oss.sgi.com Subject: fam and inetd MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-13 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.5 X-Originating-IP: 217.199.126.2 X-Virus-Scanned: ClamAV 0.80/650/Sun Jan 2 19:00:02 2005 clamav-milter version 0.80j on 127.0.0.1 X-Virus-Scanned: ClamAV 0.80/533/Sun Oct 17 04:09:44 2004 clamav-milter version 0.80j on mephisto.ktf.rtu.lv X-Virus-Status: Clean X-Virus-Status: Clean X-archive-position: 297 X-ecartis-version: Ecartis v1.0.0 Sender: fam-bounce@oss.sgi.com Errors-to: fam-bounce@oss.sgi.com X-original-sender: jancs@dv.lv Precedence: bulk X-list: fam Hi! i cant figure out such behaviour: fam is registered in inetd.conf, but when the inetd starts, it reports: inetd[204]: famd: unknown rpc service. rpcinfo -p gives 391002 2 tcp 786 sgi_fam I have to note, that this line shows up only after i manually run famd. And even then inetd does not want to recognize famd... -- Jancs Laps Cileecish Veel 293 meeneshi liidz pensijai... http://openoffice-lv.sourceforge.net http://tehvi.dv.lv ***