On Mon, 5 May 2003, Andrey Borzenkov wrote:
> > One minilogd calls bind() on /dev/log, and the other calls unlink() on the
> > same file. The problem only happens with Linux 2.4.x (not 2.5.x) and only
> > if devfs is mounted.
Yes. I could run that system with serial console dump the process list by
Break-T (the same as Alt-SysRq-T on the local console). The result is
available here:
https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=91500&action=view
It is clearly visible that there is a deadlock between sys_unlink() and
sys_bind(), which confirms my findings based on adding debug information
to minilogd.
> > Since both minilogd processes are in the "D" state, I don't think devfsd
> > is involved. They both are waiting in the kernel.
>
> I could not reproduce D state. Could you send me your stack
You see the stack. I have standard /etc/devfsd.conf without statements to
ignore /dev/log.
> Yes, I know that it involves several concurrent minilogd. Your
> patch fixes just symptom not the reason why it happens :)
Absolutely. Actually, 2.5.x kernels don't hang, but I see a lot of output
on the console on startup, maybe because minilogs fails to work correctly.
Fixing the hang in the devfs code is important but certainly insufficient.
> This is the exact problem that was fixed in my patch. What happens, is
>
> 1) first initlog is started, finds out /dev/log is unavailable,
> starts minilogd
[snip]
It would be great if you post it on bugzilla. I feel my knowledge in
this area is insufficient. Besides, it's a wrong list.
> > We have minilogd removing /dev/log without trying to find if it's in use.
> > This may or may not be OK. At least it's not OK if the kernel is not
> > fixed. That's the second problem.
>
> there is no way to find out if it is in use without race conditions.
> The only way is to try to connect but you can't be sure it won't be
> removed and recreated between connect and bind.
Properly designed API should provide means to lock an object atomically
when it's created, but UNIX sockets are decades old - it's hard to expect
them to be properly designed. Still, I hope there is a solution (although
I'm not an expert in this area).
> I do not think it is related to devfs, but who knows ... I still
> believe devfs just triggers bad minilogd behavior that was hidden
> before.
It's deadlock between devfsd_notify_de() and devfs_d_revalidate_wait().
It's pretty hard to deny that devfs is involved :-)
> I get many minilogds looping on poll and many initlogs hanging in
> connect. The fact minilogs loops on poll means it listens on SOCKET but
> never gets anything to accept. So, what is this socket all initlogs are
> trying to connect to?
Please copy it to bugzilla. I didn't write minilogd.
--
Regards,
Pavel Roskin
|