Re: A proposal for a General Clustering Framework

[Peter Badovinatz <tabmowzo@yahoo.com>]

--- Alan Robertson <alanr@unix.sh> wrote:
> Greg Lindahl wrote:
> > 
> > On Tue, Jun 05, 2001 at 01:56:37PM -0600, Alan Robertson wrote:
<snip>
> > 
> > This is good. In particular, you could imagine a simple pub/sub system
> > that would broadcast all published data, and if the local system
> > doesn't have a subscription, it discards it. Or you could have a
> > central server that distributes data only to the actual
> > subscribers. Can you write an API flexible enough to allow both
> > implementations?
> 
> The reason why I didn't worry too much about the event system was because
> it's pretty easy to write one as an application in the framework.  You could
> even make it where all events are seen by all systems in the same order...

I'd rather say it's straightforward to write, "easy" is not the right word.  In
fact, full ordering of all global events is definitely non-trivial, and gets us
into terrain other than the general events system I'm thinking about here. 
Ordering of local events is more reasonable, but in a fully general event
system "order" is an interesting question.
> 
> In fact, you could even write one pretty easily on the heartbeat API.  If
> you want events received in the same order everywhere, you might have to do
> a little more work, but it's not really that bad...
> 
> Basically, you have an event daemon on each machine.  Each of these daemons
> has knowledge of who their local clients are.

Going beyond the 'basic' cluster events (node up/down, IP address up/down), is
where the event system becomes interesting.  First assumption is a basic
plug-in style for producers and consumers.  The consumers are as described
here, they connect to a local "event" daemon.  They set up subscriptions with
that daemon for events of interest, the key is that these are both local and
cluster global events.  The daemon passes the events on.

Producers monitor the cluster (for some things the event daemon itself may be
the producer).  These are plug-ins customised to the appropriate cluster entity
for which they monitor.  They can be quiescent until a consumer decides to
request events for their target, the event daemon then 'tickles' the producer
to start monitoring.  It will then pass on information to the event daemon.
> 
> Each daemon is responsible for broadcasting events out, and notifying their
> clients of new events that they have subscribed for.
> 

Key question: filtering (for want of a better word.)  What I'm used to is that
the event daemons do the filtering:
- the subscriber passes in parameters that specify the conditions about which
they care, e.g., 'tell me if paging space used on any node is > 75%'.  In this
case,  an 'event' only occurs when page space utilisation exceeds this, prior
to its happening, a consumer sees NOTHING.
- the local event daemon distributes the request to its peers on the cluster
(or not, this can all work purely in local mode).
- each event daemon 'wakes up' the appropriate monitor for paging space.

At this point, it gets more interesting.  Does the paging space monitor:
- report EVERY change in page space utilisation?
- sample every X seconds?
  - report every sample?
  - report only when threshold exceeded?

In the IBM event manager model the producers are 'dumb'.  They do NOT know
about the event thresholds.  They report on some fixed basis, the local event
daemon decides whether to pass anything on.

Back to order.  Data is generated by independent producers, so, even on the
local node the event daemon can only generate an order that may, or may not,
match the true order of the events.

> It's just another cluster-aware app, which you can start up when/if someone
> requests it.

Yes, pretty well describes it.  But by the time you define the APIs and the
models and all that, it gets rather complex to make it fully general to allow
plug-ins for anything you care about, to handle local and global events, to try
and keep the message flow minimal and efficient, and to determine where to put
the function.
> 
> 	-- Alan Robertson
> 	   alanr@unix.sh

Peter

=====
These have been the opinions of:
Peter R. Badovinatz -- (503)578-5530 (TL 775)
wombat@us.ibm.com/tabmowzo@yahoo.com
and in no way should be construed as official opinion of 
IBM, Corp.

__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail - only $35 
a year!  http://personal.mail.yahoo.com/

Linux-cluster: generic cluster infrastructure for Linux
Archive:       http://mail.nl.linux.org/linux-cluster/