From arvidjaar@mail.ru Tue Apr 22 11:33:18 2003 Received: with ECARTIS (v1.0.0; list devfs); Tue, 22 Apr 2003 11:33:28 -0700 (PDT) Received: from f12.mail.ru (f12.mail.ru [194.67.57.42]) by oss.sgi.com (8.12.9/8.12.9) with SMTP id h3MIXGFu007548 for ; Tue, 22 Apr 2003 11:33:18 -0700 Received: from mail by f12.mail.ru with local (Exim FE.1) id 1982Zv-000E8b-00; Tue, 22 Apr 2003 22:33:15 +0400 Received: from [62.118.134.211] by koi.mail.ru with HTTP; Tue, 22 Apr 2003 22:33:15 +0400 From: "Andrey Borzenkov" To: flepied@mandrakesoft.com Cc: quintela@mandrakesoft.com, cooker@mandrakesoft.com, devfs@oss.sgi.com Subject: [PATCH] fix initlog/minilogd deadlock on /dev/log access Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [62.118.134.211] Date: Tue, 22 Apr 2003 22:33:15 +0400 Reply-To: "Andrey Borzenkov" Content-Type: multipart/mixed; boundary="----ASaCdedo-If2mk1oIOl7L2ddw:1051036395" Message-Id: X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-archive-position: 121 X-ecartis-version: Ecartis v1.0.0 Sender: devfs-bounce@oss.sgi.com Errors-to: devfs-bounce@oss.sgi.com X-original-sender: arvidjaar@mail.ru Precedence: bulk X-list: devfs ------ASaCdedo-If2mk1oIOl7L2ddw:1051036395 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit [This is crossposted because it was once reported as devfs bug. If anybody still remembers that :)] Some people reported boot hanging somewhere around "Checking modules dependencies" or "mounting filesystems". Minilogd checks for /dev/log ino/Xtime change to detect syslogd startup. Unfortunately, connect() on Unix socket modifies atime, and initlog does openlog/syslog/closelog for every line. Initlog buffers first 8K and starts output ... after the first line minilogd believes syslogd has been started and tries now to connect to /dev/log. Listening to it at the same time. I do not claim I fully understand what happens but I suspect attempting to connect to itself (because minilogd still continues to listen on /dev/log) will deadlock minilogd and next connect from initlog will hang waiting to be accepted. What is really surprising - how can it work at all ... this may indicate some sort of races in sockfs as well. the problem was associated with devfs because with disk-based /dev root is usually ro on boot i.e. /dev/log eiter cannot be created at all or cannot be modified. anyway I am pretty sure this patch fixes it because I started to have deadlock situation on every second boot. With this patch it boots now just fine. -andrey ------ASaCdedo-If2mk1oIOl7L2ddw:1051036395 Content-Type: text/x-diff; name="minilogd.patch" Content-Disposition: attachment; filename="minilogd.patch" Content-Transfer-Encoding: base64 LS0tIGluaXRzY3JpcHRzLTYuOTEvc3JjL21pbmlsb2dkLmMub3JpZwkyMDAxLTAxLTMwIDAxOjMx OjA3LjAwMDAwMDAwMCArMDMwMAorKysgaW5pdHNjcmlwdHMtNi45MS9zcmMvbWluaWxvZ2QuYwky MDAzLTA0LTIxIDIzOjAzOjI5LjAwMDAwMDAwMCArMDQwMApAQCAtMTI1LDkgKzEyNSwxNyBAQAog ICAgICAgaWYgKCAoeD4wKSAmJiAoIHBmZHMucmV2ZW50cyAmIChQT0xMSFVQIHwgUE9MTE5WQUwp KSApCiAJZG9uZSA9IDE7CiAgICAgICAvKiBDaGVjayB0byBzZWUgaWYgc3lzbG9nZCdzIHlhbmtl ZCBvdXIgc29ja2V0IG91dCBmcm9tIHVuZGVyIHVzICovCisKKyAgICAgIC8qCisgICAgICAgKiBi b3I6IGl0IHVzZWQgdG8gY2hlY2sgYXRpbWUgYXMgd2VsbC4gSXQgcHJvdm9rZWQgcmVhY2UgY29u ZGl0aW9uCisgICAgICAgKiB0aGF0IHJlc3VsdGVkIGluIGRlYWRsb2NrIGR1cmluZyBib290LiBh dGltZSBpcyB1cGRhdGVkIG9uIHNvY2tldAorICAgICAgICogY29ubmVjdC4gU28gZXZlcnkgY2Fs bCB0byBvcGVubG9nL3N5c2xvZy9jbG9zZWxvZyBjaGFuZ2VkIGF0aW1lCisgICAgICAgKiBzbyAv ZGV2L2xvZyBoYXMgYmVlbiByZWNyZWF0ZWQuIEluIHRoZSB3b3JzdCBjYXNlIGluaXRsb2cgd2Fz IGxlZnQKKyAgICAgICAqIGNvbm5lY3RlZCB0byBvbGQgL2Rldi9sb2cgd2l0aCBub2JvZHkgdG8g cmVhZCBmcm9tIGFuZCBodW5nCisgICAgICAgKi8KICAgICAgIGlmICggKHN0YXQoX1BBVEhfTE9H LCZzMikhPTApIHx8CiAJICAgIChzMS5zdF9pbm8gIT0gczIuc3RfaW5vICkgfHwgKHMxLnN0X2N0 aW1lICE9IHMyLnN0X2N0aW1lKSB8fAotCSAgICAoczEuc3RfbXRpbWUgIT0gczIuc3RfbXRpbWUp IHx8IChzMS5zdF9hdGltZSAhPSBzMi5zdF9hdGltZSkgKSB7CisJICAgIChzMS5zdF9tdGltZSAh PSBzMi5zdF9tdGltZSkgKSB7CiAJIGRvbmUgPSAxOwogCSB3ZV9vd25fbG9nID0gMDsKICAgICAg IH0K ------ASaCdedo-If2mk1oIOl7L2ddw:1051036395-- From proski@gnu.org Tue Apr 22 12:11:19 2003 Received: with ECARTIS (v1.0.0; list devfs); Tue, 22 Apr 2003 12:11:25 -0700 (PDT) Received: from fencepost.gnu.org (fencepost.gnu.org [199.232.76.164]) by oss.sgi.com (8.12.9/8.12.9) with SMTP id h3MJBIFu007857 for ; Tue, 22 Apr 2003 12:11:19 -0700 Received: from proski by fencepost.gnu.org with local (Exim 4.10) id 1983Aj-0006dP-00; Tue, 22 Apr 2003 15:11:17 -0400 Date: Tue, 22 Apr 2003 15:11:14 -0400 (EDT) From: Pavel Roskin X-X-Sender: proski@marabou.research.att.com To: Andrey Borzenkov cc: devfs@oss.sgi.com, flepied@mandrakesoft.com, quintela@mandrakesoft.com, cooker@mandrakesoft.com Subject: Re: [PATCH] fix initlog/minilogd deadlock on /dev/log access In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-archive-position: 122 X-ecartis-version: Ecartis v1.0.0 Sender: devfs-bounce@oss.sgi.com Errors-to: devfs-bounce@oss.sgi.com X-original-sender: proski@gnu.org Precedence: bulk X-list: devfs On Tue, 22 Apr 2003, Andrey Borzenkov wrote: > [This is crossposted because it was once reported as devfs bug. If > anybody still remembers that :)] > Some people reported boot hanging somewhere around "Checking modules > dependencies" or "mounting filesystems". Excellent! I tried to debug it but gave up. Actually, my experience shows that the problem happens when the framebuffer is used. Red Hat 8.0 and 9.0 with 2.4 kernels with framebuffer hang quite often. 2.5 kernels without framebuffer don't hang. I haven't used kernels without devfs for a long time, so I cannot tell if it's related. I also suspected that the time that it takes to run depmod may be related, so kernels with less modules are more susceptible. However, I later found that adding a delay to depmod won't help. Also, the problem happens on a system with the root system on ext3, but not on a system with root on reiserfs. No idea if it's related. Could you please post your patch to Red Hat Bugzilla and keep in updated? The bug is here: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=85621 > the problem was associated with devfs because with disk-based /dev root > is usually ro on boot i.e. /dev/log either cannot be created at all or > cannot be modified. I wonder if it's possible to have devfs mounted read-only on startup. > anyway I am pretty sure this patch fixes it because I started to have > deadlock situation on every second boot. With this patch it boots now > just fine. I'll check your patch as soon as I can. Hopefully this evening. -- Regards, Pavel Roskin From arvidjaar@mail.ru Wed Apr 23 07:12:12 2003 Received: with ECARTIS (v1.0.0; list devfs); Wed, 23 Apr 2003 07:12:21 -0700 (PDT) Received: from mx4.mail.ru (fallback.mail.ru [194.67.57.14]) by oss.sgi.com (8.12.9/8.12.9) with SMTP id h3NEC0Fu006228 for ; Wed, 23 Apr 2003 07:12:01 -0700 Received: from f23.mail.ru (f23.mail.ru [194.67.57.149]) by mx4.mail.ru (mPOP.Fallback_MX) with ESMTP id 375642C45A9 for ; Wed, 23 Apr 2003 17:11:18 +0400 (MSD) Received: from mail by f23.mail.ru with local id 198K0q-000Am8-00; Wed, 23 Apr 2003 17:10:12 +0400 Received: from [212.248.25.26] by win.mail.ru with HTTP; Wed, 23 Apr 2003 17:10:12 +0400 From: "Andrey Borzenkov" To: "Pavel Roskin" Cc: devfs@oss.sgi.com, flepied@mandrakesoft.com, quintela@mandrakesoft.com, cooker@mandrakesoft.com, tvigno@mandrakesoft.com Subject: Re[2]: [PATCH] fix initlog/minilogd deadlock on /dev/log access Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [212.248.25.26] Date: Wed, 23 Apr 2003 17:10:12 +0400 In-Reply-To: Reply-To: "Andrey Borzenkov" Content-Type: text/plain; charset=koi8-r Message-Id: Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by oss.sgi.com id h3NEC0Fu006228 X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-archive-position: 123 X-ecartis-version: Ecartis v1.0.0 Sender: devfs-bounce@oss.sgi.com Errors-to: devfs-bounce@oss.sgi.com X-original-sender: arvidjaar@mail.ru Precedence: bulk X-list: devfs > > > anyway I am pretty sure this patch fixes it because I started to have > > deadlock situation on every second boot. With this patch it boots now > > just fine. > > I'll check your patch as soon as I can. Hopefully this evening. > I forgot. I also disabled saving/restoring /dev/log by devfsd, do not have patch here handy but basically REGISTER ^log$ IGNORE CREATE ^log$ IGNORE CHANGE ^log$ IGNORE DELETE ^log$ IGNORE restart devfsd and remove /lib/dev-state/log This was my first suspect, but it did not fix the problem so the next step was minilogd. But logically restoring /dev/log by devfsd should trigger exactly the same problem. -andrey From proski@gnu.org Wed Apr 23 09:02:13 2003 Received: with ECARTIS (v1.0.0; list devfs); Wed, 23 Apr 2003 09:02:29 -0700 (PDT) Received: from fencepost.gnu.org (fencepost.gnu.org [199.232.76.164]) by oss.sgi.com (8.12.9/8.12.9) with SMTP id h3NG2CFu012555 for ; Wed, 23 Apr 2003 09:02:13 -0700 Received: from proski by fencepost.gnu.org with local (Exim 4.10) id 198MhH-0003V2-00; Wed, 23 Apr 2003 12:02:11 -0400 Date: Wed, 23 Apr 2003 12:02:10 -0400 (EDT) From: Pavel Roskin X-X-Sender: proski@marabou.research.att.com To: Andrey Borzenkov cc: devfs@oss.sgi.com Subject: Re[2]: [PATCH] fix initlog/minilogd deadlock on /dev/log access In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-archive-position: 124 X-ecartis-version: Ecartis v1.0.0 Sender: devfs-bounce@oss.sgi.com Errors-to: devfs-bounce@oss.sgi.com X-original-sender: proski@gnu.org Precedence: bulk X-list: devfs On Wed, 23 Apr 2003, Andrey Borzenkov wrote: > > I'll check your patch as soon as I can. Hopefully this evening. It helps on Red Hat 8.0 (tested by 4 successful reboots with the original rc.sysinit), but not on Red Hat 9. I'll have a look if anything touches /dev/log on Red Hat 9. -- Regards, Pavel Roskin