From:   Rusty Russell <rusty@linuxcare.com.au>
To:     Andrew Morton <andrewm@uow.edu.au>
Cc:     Keith Owens <kaos@ocs.com.au>,
        "netdev@oss.sgi.com" <netdev@oss.sgi.com>
Subject: Re: modular net drivers 
In-reply-to: Your message of "Thu, 22 Jun 2000 03:31:39 GMT."
             <3951889B.B4BFEA36@uow.edu.au> 
Date:   Fri, 23 Jun 2000 03:48:58 +1000
Message-Id: <20000622174858.304CE8154@halfway>
Sender: owner-netdev@oss.sgi.com
Precedence: bulk

In message <3951889B.B4BFEA36@uow.edu.au> you write:
> > I can provide code if this is still not clear how this keeps the
> > penalty for being a module in the module, and does not pollute the
> > rest of the kernel.
> 
> Please.
> 
> pseudo-code would suffice for me.

OK.  Here is how it would work:

1) struct module gets an `int cleanup_cpu'.

2) Modules supply a `deactivate()' method, which guarantees that
   (after synchronization) module count will NEVER increase.

3) Kernel thread does actual page removal (and could do execution of
   `cleanup()': whatever's easier).

4) #define __MOD_INC_USE_COUNT(mod) atomic_inc(&mod->uc.usecount)
   #define __MOD_DEC_USE_COUNT(mod)				\
	do { if (atomic_dec_and_test(&mod->uc.usecount))	\
		mod->cleanup_cpu = smp_processor_id();		\
	} while(0)

4) sys_init_module() initializes mod->uc.usecount to 1, and
   mod->cleanup_cpu to -1.

5) sys_delete_module() (probably check if mod->uc.usecount == 1):
	A) Calls mod->deactivate(),

	B) Syncs kernel (grab kernel lock, sync timers and other
	   bh()s, maybe grab entire machine if required).

	C) __MOD_DEC_USE_COUNT(mod)

6) Cleanup thread looks for modules with mod->cleanup_cpu set to
   current->processor, when it does, it can call mod->cleanup() then
   vfree() the memory.

The only limitation with this scheme is that if we lose the deactivate
race, the module could be `unloading' for an indefinite period.

Enhancements:

 We could split the module's init() function into init() and
 `activate()', where init() doesn't do anything which allows external
 code access to the module (ie. the modcount will not increase).  Then
 we can simply call `mod->activate()' and return -EBUSY if we lose the
 race.  It's probably not worth the hassle.

 Make __MOD_INC_USE_COUNT assert that mod->cleanup_cpu == -1 (you
 shouldn't be increasing the module count if you're deactivated).

 This method relies on the cleanup thread bouncing around the
 CPUs so it eventually runs on the CPU the module was removed on.
 Something more deterministic would please the pedants.

FAQ

1) Won't this be complicated to implement
	Not for my code, and I imagine not for lots of other code.

2) Why cleanup in a thread
	We may be in the module when we do the final
	MOD_DEC_USE_COUNT, so waiting until another thread is running
	on that CPU guarantees that the function has exited.

3) Won't that make kernel coding complex
	Not as much as adding struct module * to every registerable
	object.  Noone coding the kernel today should have problems
	understanding the issues here.

Clear now?
Rusty.
--
Hacking time.