Re: A proposal for a General Clustering Framework

[Alan Robertson <alanr@unix.sh>]

Michael E Brown wrote:
> 
> On Tue, 5 Jun 2001, Jeff Darcy wrote:
> 
> >
> > As for efficiency, this is where we get back to your point highlighted
> > previously.  RPC forces *gross* algorithmic inefficiencies for operations
> > needed by a cluster infrastructure.  RPC is oriented toward point-to-point
> > or broadcast communication, while the most useful conceptual structure for
> > many things in a cluster does(membership, voting, some parts of
> > heartbeating) is likely to be a ring.  Aggregating point-to-point RPC into a
> > star topology based around a temporarily-elected communications server
> > doesn't work.  Leader election is one of the services a cluster
> > infrastructure *provides*, not one it consumes from elsewhere.  The only
> > alternative implementable via RPC is all-to-all, which simply doesn't scale.
> > In short, the conceptual model of RPC forces logical-topology decisions in
> > directions that are far from optimal for clusters.
> 
> <smile>
>  Can't we all just be friends?
> </smile>
> 
> I think it would be pretty easy to define a new 'multicast' protocol spec
> to distribute XMLRPC messages. At this point, XMLRPC ceases to be RPC, but
> just a generic message passing mechanism. You can then implement the star
> model you are talking about.
> 
> The model that I am thinking about has libraries on the back-end that are
> accessible via XMLRPC through several 'transports' to remote systems. This
> gives the flexibility to use libraries in other languages that are already
> developed and in use, so we don't have to 'roll our own' protocol and
> message passing spec. Along with the 'roll our own' idea we would then
> have to write language bindings for several languages and keep them all
> up-to-date with the main library.  And _that_ isn't scalable.

AFAIK, the existing libraries require HTTP, and expect it.  Read the spec -
it *certainly* requires it.  Maybe we can write libraries in several
languages which create/extract the XML out of the "envelope" and only keep
those up to date.  This wouldn't be so bad...  If the libraries in all these
other languages have ways of bypassing the HTTP protocol and only dealing
with what the spec calls the "payload" along with information extracted from
the "envelope", then they might be of use.  If they assume HTTP they are
useful only under the surgeon's knife ;-)

Basically the envelope needs to have the following information in it when
you receive it:
	(authenticated-originator-id, authenticated-origination machine
	, destination-
	,	destlist (?))

When you send the packet, the originator only supplies the destination list
and , and the other two are filled in automatically.

I am cool with this idea except for one thing:  XML-RPC retains the worst
aspect of 'C' calling sequences - which is potentially deadly for an HA
system - positional parameters.

The usual problem with positional parameters is that they assume that
everyone is compiled against the same version of the function definition. 
Within limits, this is fine for a single non-HA system.

However, in an HA system, calls across nodes *cannot* make this assumption. 
This is because you can *never* take an HA system down all at once and
upgrade all the pieces of software at once, then bring it back up.  This is
in contradiction with the requirements of an HA system never having to go
down.

For example, let's say we have an XML-RPC function called log_message()
which takes two parameters: a format string and an array of arguments of
various types.  In version 2, we add a priority field to the call so that it
is now like this:
	log_message(priority, format, args)

With positional parameters the new code totally confuses the old code when
it the new code sends a message to the old code to log something, since its
expecting a format string as its first argument, and got an int instead.

Yuck.

If the code sent this via name/value pairs (or an XML-RPC struct), the
message would arrive with a named field it didn't understand, and all the
other arguments it could nicely understand:
	(priotity=3, format="foo", args=(1,2,3)).
Now, this is perfectly easy to code, and easy to deal with, and much more
expandable and robust, etc. than positional parameters.

This is not to say that we shouldn't use XML-RPC, but that anyone who cares
about version compatability should only send a single argument of a struct -
no matter what else you want to send...

	-- Alan Robertson
	   alanr@unix.sh

Linux-cluster: generic cluster infrastructure for Linux
Archive:       http://mail.nl.linux.org/linux-cluster/